Category Archives: Software

Software

And then it happened. I ditched Firefox

Leave a comment

The latest installment of Firefox forces linux users to use Pulseaudio. Pulseaudio is an excuse for a sound pipeline.  It does not work. It is as simple as that.

BpmDj using pulseaudio skips through the audiostream because the java audiodriver apparently cannot keep track of the pulseaudio stream. You would say: yes it it probably a java problem. Not so, many applications, also non java apps, have this problem.

https://aliver.wordpress.com/2016/02/17/why-i-dislike-pulseaudio/ has some more ranting on the state of pulseaudio. A must read.

Aside from that, pulseaudio with firefox is a complete CPU hog. Doing nothing in firefox: 47% CPU usage for the Webcontent process. Throw pulseaudio off your system: a mere 9% CPU usage (which I already find too much but you can see how pulseaudio affects my fan life).

Then, there is the problem that the mozilla developers ignore the pleas of their users wrt to this ‘feature/bug’ https://bugzilla.mozilla.org/show_bug.cgi?id=1247056 is a bloody painful read. None of the developers make any sense. It seems like they have all been lobotomized. Their arguments boil down to: ‘yeah, it was easier to program. Fuck you. And BTW, fuck you again. Please, send us your telemetry if you need more help.’ They won’t be getting much telemetry from me anymore. I ditched firefox and switched to the iridium-browser. And my goodness that thing works fast and good.

Programming, Software

A reading of a Theano compiled graph

Leave a comment

I’m trying to understand a compiled theano function, which I printed with theano.printing.debugprint. The full monster is printed below, yet it is only with a couple of lines that I have some problems.

This code first computes a random yes/no vector in node 6. Node 5 is used to create the appropriate shape (resembling x)

Gemm{inplace} [id A] <TensorType(float32, matrix)> '(dcost/dW)'   23
 |Dot22 [id B] <TensorType(float32, matrix)> ''   22
 | |InplaceDimShuffle{1,0} [id C] <TensorType(float32, matrix)> 'x_tilde.T'   12
 | | |Elemwise{Mul}[(0, 0)] [id D] <TensorType(float32, matrix)> 'x_tilde'   8
 | |   |RandomFunction{binomial}.1 [id E] <TensorType(float32, matrix)> ''   6
 | |   | |<RandomStateType> [id F] 
 | |   | |MakeVector{dtype='int64'} [id G] <TensorType(int64, vector)> ''   5
 | |   | | |Shape_i{0} [id H] <TensorType(int64, scalar)> ''   1
 | |   | | | |x [id I] <TensorType(float32, matrix)>
 | |   | | |Shape_i{1} [id J] <TensorType(int64, scalar)> ''   0
 | |   | |   |x [id I] <TensorType(float32, matrix)>
 | |   | |TensorConstant{1} [id K] <TensorType(int8, scalar)>
 | |   | |TensorConstant{0.75} [id L] <TensorType(float32, scalar)>
 | |   |x [id I] <TensorType(float32, matrix)>
 | |Elemwise{Composite{((i0 - i1) * i2 * i1)}}[(0, 2)] [id M] <TensorType(float32, matrix)> ''   21
 |   |TensorConstant{(1, 1) of 1.0} [id N] <TensorType(float32, (True, True))>
 |   |Elemwise{Composite{scalar_sigmoid((i0 + i1))}}[(0, 0)] [id O] <TensorType(float32, matrix)> 'reduced'   15
 |   | |Dot22 [id P] <TensorType(float32, matrix)> ''   11
 |   | | |Elemwise{Mul}[(0, 0)] [id D] <TensorType(float32, matrix)> 'x_tilde'   8
 |   | | |W [id Q] <TensorType(float32, matrix)>
 |   | |InplaceDimShuffle{x,0} [id R] <TensorType(float32, row)> ''   2
 |   |   |B [id S] <TensorType(float32, vector)>
 |   |Dot22 [id T] <TensorType(float32, matrix)> '(dcost/dreduced)'   20
 |     |Elemwise{Composite{((i0 * (i1 - Composite{scalar_sigmoid((i0 + i1))}(i2, i3)) * Composite{scalar_sigmoid((i0 + i1))}(i2, i3) * (i4 - Composite{scalar_sigmoid((i0 + i1))}(i2, i3))) / i5)}}[(0, 2)] [id U] <TensorType(float32, matrix)> ''   18
 |     | |TensorConstant{(1, 1) of -2.0} [id V] <TensorType(float32, (True, True))>
 |     | |x [id I] <TensorType(float32, matrix)>
 |     | |Dot22 [id W] <TensorType(float32, matrix)> ''   17
 |     | | |Elemwise{Composite{scalar_sigmoid((i0 + i1))}}[(0, 0)] [id O] <TensorType(float32, matrix)> 'reduced'   15
 |     | | |InplaceDimShuffle{1,0} [id X] <TensorType(float32, matrix)> 'W.T'   3
 |     | |   |W [id Q] <TensorType(float32, matrix)>
 |     | |InplaceDimShuffle{x,0} [id Y] <TensorType(float32, row)> ''   4
 |     | | |B_Prime [id Z] <TensorType(float32, vector)>
 |     | |TensorConstant{(1, 1) of 1.0} [id N] <TensorType(float32, (True, True))>
 |     | |Elemwise{mul,no_inplace} [id BA] <TensorType(float32, (True, True))> ''   16
 |     |   |InplaceDimShuffle{x,x} [id BB] <TensorType(float32, (True, True))> ''   14
 |     |   | |Subtensor{int64} [id BC] <TensorType(float32, scalar)> ''   10
 |     |   |   |Elemwise{Cast{float32}} [id BD] <TensorType(float32, vector)> ''   7
 |     |   |   | |MakeVector{dtype='int64'} [id G] <TensorType(int64, vector)> ''   5
 |     |   |   |Constant{1} [id BE] 
 |     |   |InplaceDimShuffle{x,x} [id BF] <TensorType(float32, (True, True))> ''   13
 |     |     |Subtensor{int64} [id BG] <TensorType(float32, scalar)> ''   9
 |     |       |Elemwise{Cast{float32}} [id BD] <TensorType(float32, vector)> ''   7
 |     |       |Constant{0} [id BH] 
 |     |W [id Q] <TensorType(float32, matrix)>
 |TensorConstant{1.0} [id BI] <TensorType(float32, scalar)>
 |InplaceDimShuffle{1,0} [id BJ] <TensorType(float32, matrix)> ''   19
 | |Elemwise{Composite{((i0 * (i1 - Composite{scalar_sigmoid((i0 + i1))}(i2, i3)) * Composite{scalar_sigmoid((i0 + i1))}(i2, i3) * (i4 - Composite{scalar_sigmoid((i0 + i1))}(i2, i3))) / i5)}}[(0, 2)] [id U] <TensorType(float32, matrix)> ''   18
 |Elemwise{Composite{scalar_sigmoid((i0 + i1))}}[(0, 0)] [id O] <TensorType(float32, matrix)> 'reduced'   15
 |TensorConstant{1.0} [id BI] <TensorType(float32, scalar)>
RandomFunction{binomial}.0 [id E]  ''   6

Then at the second marked bold section, we see that node 16 performs the multiplication of two subnodes 14 and 13; which both are very similar except for a constant 0 or 1.

The code for node 14 reuses the same vector as node 5, but then casts it to float32 (which is node 7). And then the magic, the thing I do not understand happens. From that casted vector, a subtensor with constant 0 is selected. What does this do ?

The questions:

  1. Does the SubTensor (node 10 or node 9) select the first element of the tensor from node 7 ?
  2. Node 7 is merely a casted version of node 5. Does that vector contain the random data generated in node 6 ?
  3. Once the subtensors of node 7 are selected they are allowed to broadcast (node 13 and 14) to finally be multiplied with each other in node 16. Is it correct to say that node 16 thus computes the multiplication of the first random element with the second random element (from a random vector that might be quiet somewhat larger) ?
  4. When I print out the types of the nodes, we see that the output of the subtensor is indeed a scalar (as expected), yet the type of InPlaceDimensionShuffle and the ElementWise{mul} is (True,True). What for type is that ?
  5. If ElementWise{Mul} is not specifying ‘inplace’, (as happens in node 6), which of the two children is then modified ? Is it the node associated with the RandomFunction (thus node5) or does the randomFunction (node 6) provide us with another copy that can be modified ?

The image graph of the above function is given below. The yellow box is the one that confuses me because, if I read that right its output is merely a broadcastable 0.

gradient_marked

After a day bashing my head against this nonsense I fugerd out the following:

Makevector in node 5 uses the shape as input parameters and concatenates them into a small vector (See http://deeplearning.net/software/theano/library/tensor/opt.html#theano.tensor.opt.MakeVector).

Thus its output is a vector with two values: the dimensions of x. The random generator then uses that as input to generate a correctly sized vector. And the yellow box in the computation merely multiplies the two dimensions with each other to calculate the element-count of the input. Answering each of the subquestions:

  1. yes the subtensor selects respectively the 0th and 1st element of the intput vector.
  2. node 7, does indeed contain the casted data from node 5. However node 5 does not contain the random data.
  3. it is wrong to say that node 16 computes the multiple of the two first random values. Right is: node 16 computes the multiplication of the dimension sizes of the input vector x.
  4. The (True,True) type merely tells us the broadcasting pattern. See section http://deeplearning.net/software/theano/library/tensor/basic.html#all-fully-typed-constructors
  5. Without inplace an elementwise multiplication destroys one of its inputs. The inputs that are destroyed are marked in red. documentation on color coding http://deeplearning.net/software/theano/library/printing.html#theano.printing.pydotprint
Software

How to solve memcpy error when compiling Caffe

Leave a comment

If you try to use cuda with gcc 5.0.3 according on Ubuntu 16.4 you might have the following problem:

Makefile:586: die Regel für Ziel „.build_release/cuda/src/caffe/layers/sigmoid_layer.o“ scheiterte
make: *** [.build_release/cuda/src/caffe/layers/sigmoid_layer.o] Fehler 1
/usr/include/string.h: In function ‘void* __mempcpy_inline(void*, const void*, size_t)’:
/usr/include/string.h:652:42: error: ‘memcpy’ was not declared in this scope
 return (char *) memcpy (__dest, __src, __n) + __n;

To solve this add -D_FORCE_INLINES to the following line in the Makefile:

COMMON_FLAGS += -DCAFFE_VERSION=$(DYNAMIC_VERSION_MAJOR).$(DYNAMIC_VERSION_MINOR).$(DYNAMIC_VERSION_REVISION)

The result should be:

COMMON_FLAGS += -DCAFFE_VERSION=$(DYNAMIC_VERSION_MAJOR).$(DYNAMIC_VERSION_MINOR).$(DYNAMIC_VERSION_REVISION) -D_FORCE_INLINES
Software

Is Wikipedia sustainable ?

2 Comments

As with every year: the wikimedia foundation is begging for money again. So I thought: let’s have a look at how my money would be spend.

About 1/4th of the money would be spend on administration. Not that I mind some administration but that is excessive. Let’s say they get around 20’000’000 EUR in donations.  5’000’000 EUR for administration of wikipedia is too much. The box ‘fundraising’ of course says that some fundraisers get their money back by being hired by wikimedia. In that case their ‘donation’ is no donation and should not be listed as such.

Annual report assets

A further graphic of the year 2014 shows that less than 40% of the money would be spent on actual product improvement. Around 60% goes to administration of sorts.

Annual report assets

If you are someone living in germany and you wonder how your donation will benefit the locality: it won’t do very much. Your donation is largely funneled into the US. And even less of that money is spend on product improvement: 11%.

WMDE-Mittelverwendung-2016

With the current set of employees and wikipedias consistent hunger for money, the natural question to ask is: what happens when people are tired of donating to wikipedia ? Is wikipedia sustainable ?

Will all the content be ripped from the net, slowly decay into irrelevance, filled with more spam, less content ?

I guess the only way to make wikipedia truly sustainable is by making it peer to peer. Storing all articles in a completely peer to peer manner.

And as expected if you only want to support bureacracy. Two years later: https://www.theregister.co.uk/2017/06/07/golden_handshakes_at_wikipedia/

Software

C++ sigh

1 Comment

I cannot properly express my hatred towards C++. I really cannot put it into words. You would think that starting a sentence with insults and strong words would do it, but no, it would only appear as a feeble attempt to express how I really feel. A polite ‘fuck you C++’… will only be reciprocated with a ‘fuck you too Werner’. So I’m totally at a loss. Even comparing C++ its complexities and ambiguous sense of morality to Merkel would not do it justice. Sometimes, I feel like I would like to pick up the phone, call Stroustroup, pull him through the cable and have ‘a nice and friendly chat’ with him. Will not do either because he already speaks like the Swedish cook. Maybe the best comparison would be a Kafkaesque interaction between administration trying to rob your of your will to live and you as the victim. I feel as if I want to die.

Programming, Software

The Symbian/Qt/Nokia platform. A total nightmare.

Leave a comment

In June/July 2011 I had the misfortune to program for a Symbian/Nokia phone. Really bad experience. Below  a list of all the shit I had to go through to get QtCreator working somewhat.

  • crappy error messages. ‘a data abort exception has occured’
  • qml is a complete paradigm shift
  • unstable operating system. ‘it might be in an unstable state if you interrupt the working of this program’
  • version control unfriendly. Often a file added by somebody else must be readded in the repository which leads to duplicate entries in the project file.
  • memory mapping of large files doesnt work
  • memory requirements are too stringent
  • debugger doesn’t work. One always gets memorydumps without access to the symbols.
  • bloody slow compilation
  • the X7 Symbian 3 phone is slow.
  • Try to get the audiostream from a decoder. With phonon it should be possible but it doesnt work.
  • just click on the ‘documentation’ tab left. You get an error ‘Die Seite konnte nicht gefunden werden ‘qthelp://com.nokia.qtcreator.220/qdoc//doc/index.html
  • sometimes code gets hidden and cannot be unhidden unless the project is opened and closed.
  • how can one create a subfolder ?
  • copying files to and from the phone more often than not does not work.
  • try to use phonon or qtmultimedia to access the decoded part of a video !
  • searching in all project files simply doesn’t work.
  • qt creator often seems to require compilation twice.  ‘build’ and then ‘run’, leads to a rebuild.
  • no linux, nor vmware. You must program in windows
  • QtCreator crashes when there are more than 4000 error messages from the compiler.
  • sometimes when adding a new file, it insists to add it somewhere in a subdirectory.
  • The arm processor in those phones is not fully functional. For instance the ‘smull’ and ‘adc’ instructions are not supported. Mainly because it is based on a too old model of the processor.
  • ‘piss-poor’ headphone volume; something many buyers noted as well
  • Filenames (without their paths) are used to identify files. So the same filename at twop different locations is treated by the make system as the same file.
  • How to set the volume of th phone ?  Yes a truly horendous hack by obtaining the address of a undocumented function and then hoping that it will actually point to the right thing ! http://bugreports.qt.nokia.com/browse/QTMOBILITY-830
  • No support from Nokia what_so_ever. Even if you are hired by them (which I inderectly was !)

I was so happy that I could stop programming for/on this utter swamp of a platform.

Software

Synchronizing a wiki between 2 machines

Leave a comment

Getting a wiki from the server

#!/bin/bash
if [ -e wiki_is_local ]; then
echo "Wiki is already locally instantiated"
exit
fi
echo "Downloading files"
rsync -xavz --exclude LocalSettings.php root@sigtrans.org:/home/nens/wiki /home/nens/
echo "Downloading database"
mysqldump -C --host=213.239.213.249 --user=NensWiki --password=passwd NensWiki >server_state.sql
echo "Inserting in local database"
mysql -h localhost -u NensWiki --password=passwd -D NensWiki

Putting a wiki towards the server

#!/bin/bash
echo "Creating local copy of database"
mysqldump -C --host=localhost --user=NensWiki --password=passwd NensWiki >local_state.sql
echo "Synchronizing mediawiki directory"
rsync -xavz --exclude LocalSettings.php /home/nens/wiki root@sigtrans.org:/home/nens/
echo "Synchronizing remote database"
mysql -C -h 213.239.213.249 -u NensWiki --password=passwd -D NensWiki
Software

Closed Source Drivers & The Linux Community

Leave a comment

Every time I come in contact with a closed source ‘proprietary’ driver I find that it offers me more trouble than it is worth.

NVidia

Nvidia is the kind of cooperation that pollutes the Linux community. Linux is open source, the Nvidia drivers are not. They are proprietary, which is silly in general. I buy hardware and when I buy hardware I want a pretty decent specification on how to use it (e.g: under the form of source code). If that is not available, then I bought something useless. The NVidia drivers, and specially the 3D access, are closed source and incredible difficult to install. I managed to do this for all past kernels and versions of their closed source driver. However very recently my card was ‘suddenly’ no longer supported. This means that I now have the choice of buying a new one (probably the big master plan behind their current driver), or fall back to the less-than-optimal X.org implementation of the NVidia driver. I ‘choose’ the last option.

ATI Radeon cards: Fglrx

The firegl cards are a nightmare as well. Again a closed sourced driver and a company that tries to get their ‘hands’ in the market by not releasing the source of their driver in one way or another. These days this might seem ‘natural’ not to provide a specification of hardware. But, the reality is that one effectively buys a car without the ability to steer it. For instance it has taken many generations of the Radeon driver to actually not crash X every once in a while (probably a concurrency problem). However, it was completely impossible to fix that issue since no source was available. Similarly: lately the X.org drivers changed their versioning scheme which broke the binary driver entirely. Normally one could fix that, now we can’t.

It finally happened. One of our machines stopped working with the upgrade to the 8.38 drive. The mobility Radeon 9000 is no longer supported. So here we are: great hardware, no drivers. Does this mean their hardware is completely useless ? The answer is: yes.

Sound Blaster

Sound blaster cards and creative labs in general are a nightmare for the Linux community. In a sense they are partly supported, in another sense they are supported because some people have spent a lot of time in figuring out how the cards work. However, what I can now do with my ‘Create Labs live mega DSP piece of hardware’ is that I can access the DA and AD circuits. That’s about it. There are simply no proper specifications available on how to access the hardware.

Intel IPW2200 drivers

At the moment I have a Intel IPW2200 wireless network card. That driver is closed source as well, and I’m pretty sure that in the near future the drivers will become obsolete as well.

Software

Adobe Flash 9 Audio Ticks in Firefox Linux

Leave a comment

I had an annoying problem with flash under firefox: ticks during playback. The solution was to create an alsa soundrc file with the following content.

pcm.main {
type hw
card 0
}ctl.main {
type hw
card 0
}

pcm.!default {
type dmix       # dmix plugin for mixing the output
ipc_key 1234    # a unique number
slave {
pcm “main”
period_time 0
period_size 8192
buffer_size 32768
rate 44100
}
}

The above chunk oif text should be placed in /etc/asound.conf or in ~/.asoundrc, depending on whether you want this setup to be system wide or for you own account only.

There are a number of important points:
– this was also true for alsalib version 1.2.10
– the key is the period_size and buffer_size
– flash cannot be configured to use another audiodevice, so the default is important.
– don’t forget competing drivers to the soundcard. That is: kill jackd, artsd and other blahd’s

Software

Using Electric Fence to Debug Selected Allocations

Leave a comment
Electric Fence is a Red-Zone memory allocator written by Bruce Perens. It provides a special version of malloc() and similar functions for debugging software that is suspected of overrunning or underrunning the boundaries of a malloc buffer, or touching free memory. It arranges for each malloc buffer to be followed (or preceded) in the address space by an inaccessable virtual memory page, and for free memory to be inaccessable. If software touches the inaccessable page, it will get an immediate segmentation fault. It is then trivial to uncover the offending code using a debugger. An advantage of this product over most malloc debuggers is that this one detects reading out of bounds as well as writing, and this one stops on the exact instruction that causes the error, rather than waiting until the next boundary check. A Secondary advantage is that it can be used as a replacement of the standard malloc, thereby debugging software that cannot be recompiled or tracking allocations within libraries themselves.

 

This however forms immediatelly the problem I encountered. When debugging my pet-project BpmDj, I found that the QT libraries use extravagant many memory allocations, making the use of electric fence impossible, due to its memory consumption (2 pages = 16K for every single allocation !). Nevertheless, the thing I wanted to do was to check the memory chunks that I had allocated, not those allocated by others. Below we explain how this can easily be done with electric fence.

Step 1: change all malloc’s, realloc’s and frees in your software with own versions. E.g: allocate, reallocate, deallocate. Prototypes of such functions can be found in the file common.h

#define allocate(size, type)          (type*) bpmdj_alloc(sizeof(type)*(size), __FILE__, __LINE__)
#define array(name,size,type)          type*  name = allocate(size,type)
#define reallocate(thing, size, type) (type*) bpmdj_realloc(thing,sizeof(type)*(size))
#define deallocate(thing) bpmdj_free(thing);void * bpmdj_alloc(int size, char* file, int line);
void * bpmdj_realloc(void* thing, int size);
void   bpmdj_free(void*);

Step 2: modify the library to NOT replace the standard memory operations. The only files needed from the electric fence package are efence.h page.c, efence.c and efence-print.cpp

Step 2.1: modify all references to malloc, calloc, realloc and free in efence.c to efence_malloc, efence_calloc, efence_realloc and efence_free. Make sure that prototypes for these functions are added into efence.h.


void * efence_realloc(void * oldBuffer, size_t newSize)
{
    void *    newBuffer = efence_malloc(newSize);
…
        efence_free(oldBuffer);
…
}void * efence_malloc(size_t size)
{
…
}

void * efence_calloc(size_t nelem, size_t elsize)
{
…
}

Step 2.2: modify the efence_free function to return a boolean. If the freeing was successfull true should be returned. If not because the pointer itself was not within range of the efence allocated memory thenen it should return false. This will later prove to be very usefull to write a generic deallocate function.

bool efence_free(void * address)
{
…
  if ( address == 0 )
    {
      unlock();
      return true;
    }
…
  if ( !slot )
    {
      unlock();
      return false;
    }
    // removed EF_Abort(“efence_free(%a): address not from efence_malloc().”, address);
…
  return true;
}

Step 2.3: the standard efence-print library makes ues of its own printing functions. These can be safely removed because the standard C library functions no longer pose a reentrance thread. Thus, remove printNumber and vprint. Replace all vprint calls with vprintf and include the necessary #include <stdlib.h>

Step 2.4: change efence-print.cpp, efence.cpp, page.cpp, efence.h by putting an #ifdef EFENCE in front and an #endif in the back of the file. This will make it possible to remove the entire EFENCE from compiling.

Step 3: write your own memory functions as declared in step 1. These are straightforward, except for the deallocate function. Here we must differentiate between efence-allocated chunks and an standard-malloced chunks. This can be done as follows.

void bpmdj_free(void* a)
{
#ifdef EFENCE
if (!efence_free(a))
#endif
free(a);
}void* bpmdj_alloc(int length, char* file, int line)
{
void * result;
assert(length>=0);
#ifdef EFENCE
result = efence_malloc(length);
#else
result = malloc(length);
#endif
if (!result)
printf(“Error: %s(%d): unable to allocate %d bytes \n”,file,line,length);
assert(result);
return result;
}

void* bpmdj_realloc(void* thing, int size)
{
void * result;
assert(size);
#ifdef EFENCE
result = efence_realloc(thing,size);
#else
result = realloc(thing,size);
#endif
assert(result);
return result;
}

This should do the trick. To use it simply put the object files efence.o, page.o, efence-print.o and common.o into your program. When you want checks pass -DEFENCE to the compiler, in the other case simply ommit it. These files can be downloaded here. This should make it possible to debug QT applications easily under linux.