Category Archives: Programming

.NET Obfuscators

BpmDj… my brainchild… bringing in no money… So I am looking for free tools.

In order to figure out how much information assemblies throw around,  just have a look at https://www.jetbrains.com/decompiler/ It basically returns me the original source code, including all variable names and everything else that could have been thrown out. Therefore, an obfuscator is really necessary. http://www.dotnetstuffs.com/best-free-obfuscator-to-protect-net-assemblies/ had a list of interesting possibilities.

Dotfuscator (A Lead to Sell)

The microsoft site refers to ‘DotFuscator’; and let me tell you.. the community edition is bullshit. The entire thing is one big lead to sell you their product. It starts with a forced registration (you have to give a valid emailaddress). Then when you are in the application, you only see advertisment, not a lot of real useful obfuscation going on. And lastly, when I ran it on BpmDj it wasn’t even able to go through it because ofg ‘mixed assemblies’ I am sure I could set up a joint project with preemtive solutions, in which I would of course pay them, but honestly… don’t bother with this bullshit. The community edition doesn’t do what it pretends it will do.

Obfuscar (No XAML)

Obfuscar tries to map as many input names to as few output names as possible. ‘Massive overloading’ as they call it. https://code.google.com/archive/p/obfuscar/

At first glance this seems a dead end.. Last release was 11 years ago. However stack overflow posts still discuss it in 2018. Ha no, it seems to have moved to https://github.com/obfuscar/obfuscar

Amazingly enough, after getting a simply configuration it actually ran through the entrire shebang of assemblies and generated 1 output. That output could even start ! Yet it did hang at the splashscreen. Attaching a debugger showed that all threads had been started properly, so I assume either I access a dynamic resource by name (I do have some explicit invokes laying around), or the XAML bindings were seriously fucked up. This is something I should test somewhat further, because if this works we are done.

Oh the horror. The default configuration doesn’t actually obfuscate shit. All identifiers were still present, despite the fact that it claims it had a ‘mapping’. Probably it kept all public properties public as they were without renaming them.

A list of options for the Obfuscar xml file

InPath = Environment.ExpandEnvironmentVariables(vars.GetValue("InPath", "."));
 OutPath = Environment.ExpandEnvironmentVariables(vars.GetValue("OutPath", "."));
 LogFilePath = Environment.ExpandEnvironmentVariables(vars.GetValue("LogFile", ""));
 MarkedOnly = XmlConvert.ToBoolean(vars.GetValue("MarkedOnly", "false"));

RenameFields = XmlConvert.ToBoolean(vars.GetValue("RenameFields", "true"));
 RenameProperties = XmlConvert.ToBoolean(vars.GetValue("RenameProperties", "true"));
 RenameEvents = XmlConvert.ToBoolean(vars.GetValue("RenameEvents", "true"));
 KeepPublicApi = XmlConvert.ToBoolean(vars.GetValue("KeepPublicApi", "true"));
 HidePrivateApi = XmlConvert.ToBoolean(vars.GetValue("HidePrivateApi", "true"));
 ReuseNames = XmlConvert.ToBoolean(vars.GetValue("ReuseNames", "true"));
 UseUnicodeNames = XmlConvert.ToBoolean(vars.GetValue("UseUnicodeNames", "false"));
 UseKoreanNames = XmlConvert.ToBoolean(vars.GetValue("UseKoreanNames", "false"));
 HideStrings = XmlConvert.ToBoolean(vars.GetValue("HideStrings", "true"));
 Optimize = XmlConvert.ToBoolean(vars.GetValue("OptimizeMethods", "true"));
 SuppressIldasm = XmlConvert.ToBoolean(vars.GetValue("SuppressIldasm", "true"));

XmlMapping = XmlConvert.ToBoolean(vars.GetValue("XmlMapping", "false"));
 RegenerateDebugInfo = XmlConvert.ToBoolean(vars.GetValue("RegenerateDebugInfo", "false"));

At least when setting the KeepPublicApi to false, some obfuscation happened.

https://stackoverflow.com/questions/28031859/rename-an-internal-class-with-obfuscar has a nice example of a config file. Now, from what I see, the BAML is used to quickly figure out which elements of the UI refer to classes by name. That is obviously smart, yet I would like to rename the XAML entries as well.

I also figured out that the BAML/XAML tree is still stored in the assembly as it was present in the original solution, so no reordering takes places in any way. Not a total failure, but not great either because of this.

ConfuserEx (Abandonded)

Finally something that is not a landing page. Huray ! Last update… 26 January 2019.. still it might work and it is open source. The projecct has indeed be discontinued since 1. July 2016.

In any case, a run of it did behave similarly as Obfuscar. The application started and didn’t get further,probably because of the missing DLL’s. I might need to fix that problem if no obfuscator gets through it. In any case, the XAML was effectively gone after obfuscating; or the dotPeek decompiler stopped trying. I am not entirely sure what it is yet.

After spending some hours on this problem, the problem seems to be in the renaming strategy used. I am not yet sure whether I will blame confuserex or my own program, given that Obfuscar had exactly the same error as this one. Then again maybe they both are based on the same source, so they might both be suffering from the same bug.

In any case, performing a ‘none’ protection did not damage the original assembly, which is already a good sign. Also nice was that there was a debug protection in place which caused the applciation to bark when a debugger tries to connect.

Skater Light (Does not  obfuscate at all)

Also Skater is a piece of software to buy. They do have a free version, named SkaterLight. Oddly enough… this feels a lot like chinese spyware. Seriously.

  1. it worked at the first attempt. So I was a bit skeptical… I decompiled the generated assembly and lo and behold the thing was just not obfuscated whatsoever.
  2. after installing it, it actually ran with elevated privileges (I know that because I could not read the generated assembly)
Yes… ‘Full Obfuscation’
Just don’t believe it

Dead Ends

  • Eazfuscator (not free) – Next is Eazfuscator because they seem very eager to actually deal with the WPF/XAML issue. Oh well.. forget it. Not free anymore. This is the point where I considered whether it would be possible to use a decompile tool to decompile an obfuscator, remove the licensing restrictions and continue. There is a certain beauty to this approach: if the obfuscator sucks, then we can easily do that, which makes it pointless to actually use then
  • CodeFort (disappeared) – was mentioned as another option which works well with XAML. Yet, latest udpate on the twitter feed was 2010 and the domain itself became a lnading page.
  • Agile.NET (non free)
  • FXProtect (disappeared)
  • ILProtector (not free) – has gone commercial since version 2.0.17
  • Babel.NET (not free) – https://www.babelfor.net/downloads/
  • SharpObfuscator (abandonded 2007) – was at https://archive.codeplex.com/?p=sharpobfuscator 
  • Goliath.NET (not free)
  • .NET Reactor (not free) Nice page
  • .NETGuard (not free)
  • Smart Assembly (excessive pricing)
  • CodeVeil – encrypts the DLL before executing it. In the end this might be a better option than ‘obfuscating’ it. Drawback is of course that we have a single point of failure. Another drawback is that it is a chinese product and only a trial version.
  • CryptoObfuscator (not free)
  • Rummage (fair pricing)
  • Xenocode (abandonded)
  • DeepSea Obfuscator
  • MaxtoCode
  • MPRESS
  • Spices.Net

The link https://github.com/de4dot/de4dot/tree/master/de4dot.code/deobfuscators is interesting because it lists existing obfuscators 1) that can be deobfuscated 2) were sufficiently know to bother deobfuscating 3) how difficult each one is

A new object cache for BpmDj

In BpmDj we load objects on demand: every time a particular object is accessed we load it from the database. This process happens automatically, and is implemented through a dictionary which maps an object id to a runtime representation.

In Java, this dictionary was a WeakDictionary, which is a dictionary from which values can be removed by the garbage collector. When when they got removed and the program accessed that object again, we would load it fresh from the database. This poor man caching is not particularly good because any garbage collect will remove all loaded (but unreferenced) objects, forcing the program to reload those object again. Even if the particular object is often used.

To solve that, we could force references to stay in memory by means of a round robin queue. Every time an object is accesed it is put in the next position in the buffer. As such, we ensure that the cache keeps X instances alive.

Sadly that strategy is unable to deal with a burst of requests. Any often used object will simply be pushed out of the buffer when a batch of new objects is loaded (like for instance when the song selector opens).

To alleviate this problem, we can, with each access, gradually increase the stickiness of a cache item. This idea turned out to be fairly efficient:

  • every entry has a position in the buffer. Whenever the entry is hit, it moves to half its original position.
  • every new element is placed in the middle of the buffer.

This strategy leads to a distribution where often used elements are in front of the buffer. Lesser used elements slowly walk their way out of the buffer until they are evicted. To avoid that items become too sticky (e.g: there can be items that have just enough been accessed to never leave the buffer again), it is useful to add a random element to this

  • reposition an element to a random position between 0 and ratio * originalRank.

One could argue that having too many object id’s and too few actual objects would be a cause of concern, and it clearly is. Nevertheless, there often is a space tradeoff between holding on to an object and using its id.

The image shows the buffer of a cache of capacity 100, with 800 distinct element randomly accessed. The access pattern was shaped according to a power law distribution. The front of the cache are those that are more sticky than the later part of the buffer. The height of each entry indicates its priority in the emitter.

The following picture shows the difference between 3 types of cache. The first is the roundrobin mentioned earlier, the second is a cache which keeps backreferences and the elevator cache is the one implemented here.

The data on which this was ran was the retrieveal of all startup objects BpmDj need, including the opening of the song selector. The total object count was 133632, of which 70291 unique ones.

XUnit vs MSTest

After having tested both of them extensively I can draw the following conclusion: MSTest is definitely the winner. Why ?

XUnit

  • buggy as hell. For a testframework this is kinda weird
  • very very slow
  • really confused about the tests that are available
  • No standard output. Yes I know you can redirect it, still they should not steal my debug output in the first place.
  • Different assertions than MSTest, and they are badly implemented at that (E.g: an assertion finding the content of a collection will simple iterate over all elements. It is truly painful to see how far computer scientists have sunken)
  • crashes VS2019 when in auto-hide
  • Talks about [Theories] and [Facts] instead of [TestMethod], just some ‘cool’ jargon and indeed far removed from reality.

MSTest

Does not have the same level of ‘we are so cool but can’t program’ fuckery as Xunit

Allthough this post is small, nobody seems to care to say how bad xUnit exactly is.

Scoping rules in WPF

Staticresources are resolved at parsing time

A staticresource is read when the xaml is read. This is demonstrated in the following example

<Style TargetType="{x:Type TextBlock}">
 <Setter Property="Foreground" Value="{StaticResource NormalTextColor}"/>
 </Style>

<SolidColorBrush x:Key="NormalTextColor" Color="Yellow"></SolidColorBrush>

At the moment the style is created, NormalTextColor is not defined yet. And so it stays whenever later that style is applied. If we swap the NormalTextColor definition and the Style then it will be a fixed yellow.

Dynamicresources are resolved whenever necessary

If we modify the StaticResource in a DynamicResource, then that example will behave correctly, and every textblock will have a yellow foreground.

Resource Lookup

The resource lookup goes from child to parent.

<Window x:Class="WinWpfTests.MainWindow"
 Title="MainWindow" Height="450" Width="800">
 <Window.Resources>
   <SolidColorBrush x:Key="NormalTextColor" Color="Orange"></SolidColorBrush>
 </Window.Resources>
 <StackPanel Grid.Row="1" Orientation="Horizontal">
   <StackPanel.Resources>
     <SolidColorBrush x:Key="NormalTextColor" Color="Red"></SolidColorBrush> 
   </StackPanel.Resources>
 <TextBlock Text="Some text"/>
 </StackPanel>
</Window>

This means that any textblock within the stackpanel will be colored red, while the textblocks outside the stackpanel yet inside the window will be orange. And ifg the application.xaml is defined as in our first example, then any other window will be yellow.

It might be necessary to restyle multiple controls

Whenever a textblock is used it will have the provided style. A label however has its own foreground color defined, and so requires an extra style.

<Style TargetType="{x:Type TextBlock}">
 <Setter Property="Foreground" Value="{DynamicResource NormalTextColor}"/>
 </Style>
<Style TargetType="{x:Type Label}">
 <Setter Property="Foreground" Value="{DynamicResource NormalTextColor}"/>
 </Style>

Resource lookup through templates

ControlTemplates provide a way to render a particular element differently.  The  dynamic lookup still goes from the lexical point of insertion up the logical tree. Thus the following fragment

 <Style x:Key="ModifiedLabel" TargetType="{x:Type Label}">
 <Setter Property="Template">
 <Setter.Value>
 <ControlTemplate TargetType="{x:Type Label}">
 <StackPanel Orientation="Vertical">
 <TextBlock Text="Zhe legend"/>
 <ContentPresenter/>
 </StackPanel>
 </ControlTemplate>
 </Setter.Value>
 </Setter>
 </Style>

Will render both the ‘Zhe legend’ as well as the actual content of the label using the same dynamicresource: that is they will both have the same color, even if the controltemplate was defined in a different file. (One could expect that ‘Zhe Legend’ would follow a lookup hierarchy going from the definition of the template, while the ContentPresenter would follow a different hierarchy)

The logical parent with controltempaltes

The logical parent of the contentpresenter is the controltemplate, which is the same as the control being templated. Thus if we set the template of a label to something, and then define a resources in the controltemplate (as ControlTempalte.Resources), then these resources are part of the label, and thus are visibly to dynamicresources applied to the contentpresenter.

Yet, if we place the resources to a subelement within the controltemplate, then they are not part of the label, and thus not part of the logical chain of parents from the contentpresenter.

Under the assumption that the default color has been set to red in the App.xaml, we have two ways to define a controltemplate, with two different results

 

Who has priority ?

 

Because both the ControlTemplate and the original instantiation both access the same resource dictionary frame, it is useful to figure out who has priority. The answer is: the controltempalte its resources are applied first, afterwards those defined in the actual instantiation of the control.

Shortest Distance from a Point to a Line segment

A straight translation from python to java; taken from  http://www.fundza.com/vectors/point2line/index.html

public double distance2(Point2D pnt)
  {
    double lineVecDx=to.x-from.x;
    double lineVecDy=to.y-from.y;
    double pntVecDx=pnt.getX()-from.x;
    double pntVecDy=pnt.getY()-from.y;
    double lineLen=dist(lineVecDx,lineVecDy);
    double pntVecLength=dist(lineVecDx,lineVecDy);
    double lineUnitvecDx=lineVecDx/pntVecLength;
    double lineUnitvecDy=lineVecDy/pntVecLength;
    double pntVecScaledDx=pntVecDx/lineLen;
    double pntVecScaledDy=pntVecDy/lineLen;
    double tx = lineUnitvecDx * pntVecScaledDx;
    double ty = lineUnitvecDy * pntVecScaledDy;
    double t = tx+ty;
    if (t<0) t=0;
    else if (t>1) t=1;
    double nearestX=lineVecDx*t;
    double nearestY=lineVecDy*t;
    return dist(nearestX-pntVecDx, nearestY-pntVecDy);
  }

double dist(double dx, double dy)
  {
    return Math.sqrt(dx*dx+dy*dy);
  }

A replacement for the JavaFx FontMetrics class for JDK9

Most people used an internal javafx FontMetrics class, which has been deprecated in version 9 of the jdk. That means that your app that relied on this will simply not work anymore. Below is a simple replacement that will provide the computeStringWidth as well as ascent, descent and lineHeight. The produiced values are exactly the same as if they were called from the FontMetrics class itselve.

import javafx.geometry.Bounds;
import javafx.scene.text.Font;
import javafx.scene.text.Text;
public class FontMetrics
{
 final private Text internal;
 public float ascent, descent, lineHeight;
 public FontMetrics(Font fnt)
 {
 internal =new Text();
 internal.setFont(fnt);
 Bounds b= internal.getLayoutBounds();
 lineHeight= (float) b.getHeight();
 ascent= (float) -b.getMinY();
 descent=(float) b.getMaxY();
 }

 public float computeStringWidth(String txt)
 {
 internal.setText(txt);
 return (float) internal.getLayoutBounds().getWidth();
 }
}

JavaFx 8 command line options

I have been struggling with an “interesting” javafx render problem. In BpmDj, given enough updates, the rendertree would partially stop updating. It was clear this was a concurrency bug, yet one that was not my fault. As a preliminary solution to fix this I would grab a screenshot everytime I would add a node. Recently however that solution no longer worked and the dirty logic would be so fucked up that elements which should be visible were not visible. It was even possible to have a font change its color halfway through its rendering.

So I started to look around for javafx options, and there are a few of them that proved to be useful.

What Flag Default
VSync prism.vsync true
Dirty region optimizations prism.dirtyopts true
Occlusion Culling prism.occlusion.culling true
dirtyRegionCount prism.dirtyregioncount 15
Scrolling cache optimization prism.scrollcacheopt false
Dirty region optimizations prism.threadcheck false
Draws overlay rectangles showing where the dirty regions were prism.showdirty false
Draws overlay rectangles showing not only the dirty regions, but how many times each area within that dirty region was drawn (covered by bounds of a drawn object). prism.showoverdraw false
Prints out the render graph, annotated with dirty opts information prism.printrendergraph false
Force scene repaint on every frame prism.forcerepaint false
disable fallback to another toolkit if prism couldn’t be init-ed prism.noFallback false
Shape caching optimizations prism.cacheshapes complex
New javafx-iio image loader prism.newiio true
Verbose output prism.verbose false
Prism statistics print frequency, <=0 means “do not print” prism.printStats 0
Debug output prism.debug false
Trace output prism.trace false
Print texture allocation data prism.printallocs” false
Disable bad driver check warning prism.disableBadDriverWarning” false
Force GPU, if GPU is PS 3 capable, disable GPU qualification check. prism.forceGPU false
Skip mesh normal computation prism.experimental.skipMeshNormalComputation false
Which driver to use prism.order
prism.forcepowerof2 false
prism.noclamptozero false
Try -Dprism.maxvram=[kKmMgG] prism.allowhidpi true
prism.maxvram 512 * 1024 * 1024
Try -Dprism.targetvram=[kKmMgG]|<double(0,100)>% prism.targetvram
prism.poolstats false
prism.pooldebug false
prism.maxTextureSize
prism.minrttsize
prism.disableRegionCaching
prism.disableD3D9Ex false
prism.disableEffects false
prism.glyphCacheWidth 1024
prism.glyphCacheHeight 1024
Enable the performance logger, print on exit, print on first paint etc. sun.perflog
sun.perflog.fx.exitflush
sun.perflog.fx.firstpaintflush
sun.perflog.fx.firstpaintexit
prism.supershader true
Force uploading painter (e.g., to avoid Linux live-resize jittering) prism.forceUploadingPainter false
Force the use of fragment shader that does alpha testing (i.e. discard if alpha == 0.0) prism.forceAlphaTestShader false
Force non anti-aliasing (not smooth) shape rendering prism.forceNonAntialiasedShape false
Set Single GUI Threading quantum.singlethreaded false
Print quantum verbose quantum.verbose false
JavaFx framerate in FPS javafx.animation.pulse 60

FX thread collision with Render thread ?

When running with the prism.threadcheck option I got the following error:

ERROR: PrismPen / FX threads co-running: DIRTY: false
FX: java.lang.Thread.getStackTrace(Thread.java:1559)
FX: com.sun.javafx.tk.quantum.QuantumRenderer.checkRendererIdle(QuantumRenderer.java:247)
FX: com.sun.javafx.tk.quantum.QuantumToolkit.checkFxUserThread(QuantumToolkit.java:424)
FX: javafx.scene.Scene$MouseHandler.process(Scene.java:3680)
FX: javafx.scene.Scene$MouseHandler.access$1500(Scene.java:3485)
FX: javafx.scene.Scene$MouseHandler$1.run(Scene.java:3521)
FX: com.sun.javafx.application.PlatformImpl.lambda$null$173(PlatformImpl.java:295)
FX: java.security.AccessController.doPrivileged(Native Method)
FX: com.sun.javafx.application.PlatformImpl.lambda$runLater$174(PlatformImpl.java:294)
FX: com.sun.glass.ui.InvokeLaterDispatcher$Future.run(InvokeLaterDispatcher.java:95)
FX: com.sun.glass.ui.gtk.GtkApplication._runLoop(Native Method)
FX: com.sun.glass.ui.gtk.GtkApplication.lambda$null$49(GtkApplication.java:139)
FX: java.lang.Thread.run(Thread.java:748)
QR: com.sun.javafx.sg.prism.NGCanvas.getStroke(NGCanvas.java:777)
QR: com.sun.javafx.sg.prism.NGCanvas.setupStroke(NGCanvas.java:785)
QR: com.sun.javafx.sg.prism.NGCanvas.handleRenderOp(NGCanvas.java:1212)
QR: com.sun.javafx.sg.prism.NGCanvas.renderStream(NGCanvas.java:1097)
QR: com.sun.javafx.sg.prism.NGCanvas.renderContent(NGCanvas.java:606)
QR: com.sun.javafx.sg.prism.NGNode.doRender(NGNode.java:2053)
QR: com.sun.javafx.sg.prism.NGNode.render(NGNode.java:1945)
QR: com.sun.javafx.sg.prism.NGGroup.renderContent(NGGroup.java:235)
QR: com.sun.javafx.sg.prism.NGRegion.renderContent(NGRegion.java:576)
QR: com.sun.javafx.sg.prism.NGNode.doRender(NGNode.java:2053)
QR: com.sun.javafx.sg.prism.NGNode.render(NGNode.java:1945)
QR: com.sun.javafx.sg.prism.NGGroup.renderContent(NGGroup.java:235)
QR: com.sun.javafx.sg.prism.NGRegion.renderContent(NGRegion.java:576)
QR: com.sun.javafx.sg.prism.NGNode.doRender(NGNode.java:2053)
QR: com.sun.javafx.sg.prism.NGNode.render(NGNode.java:1945)
QR: com.sun.javafx.sg.prism.NGGroup.renderContent(NGGroup.java:235)
QR: com.sun.javafx.sg.prism.NGRegion.renderContent(NGRegion.java:576)
QR: com.sun.javafx.sg.prism.NGNode.doRender(NGNode.java:2053)
QR: com.sun.javafx.sg.prism.NGNode.render(NGNode.java:1945)
QR: com.sun.javafx.sg.prism.NGGroup.renderContent(NGGroup.java:235)
QR: com.sun.javafx.sg.prism.NGRegion.renderContent(NGRegion.java:576)
QR: com.sun.javafx.sg.prism.NGNode.doRender(NGNode.java:2053)
QR: com.sun.javafx.sg.prism.NGNode.render(NGNode.java:1945)
QR: com.sun.javafx.sg.prism.NGGroup.renderContent(NGGroup.java:235)
QR: com.sun.javafx.sg.prism.NGRegion.renderContent(NGRegion.java:576)
QR: com.sun.javafx.sg.prism.NGNode.doRender(NGNode.java:2053)
QR: com.sun.javafx.sg.prism.NGNode.render(NGNode.java:1945)
QR: com.sun.javafx.tk.quantum.ViewPainter.doPaint(ViewPainter.java:477)
QR: com.sun.javafx.tk.quantum.ViewPainter.paintImpl(ViewPainter.java:330)
QR: com.sun.javafx.tk.quantum.PresentingPainter.run(PresentingPainter.java:91)
QR: java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
QR: java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
QR: com.sun.javafx.tk.RenderJob.run(RenderJob.java:58)
QR: java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
QR: java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
QR: com.sun.javafx.tk.quantum.QuantumRenderer$PipelineRunnable.run(QuantumRenderer.java:125)
QR: java.lang.Thread.run(Thread.java:748)

Of course how that related to my problem wasn’t clear, allthough maybe it could explain why the dirty flags were all ****ed up.

Overdraw and dirty draws

overdraw

The option -Dprism.showoverdraw=true is interesting because it shows us which area we drew once (red), which twice (green), and so on. This showed me that the areas that didn’t update were not even marked as ‘dirty’. Suggesting that the problem was indeed with the setting and tracking of the dirty flag in the renderer. Therefore we tried the option -Dprism.dirtyopts=false and hooray all problems disappeared. Of course the app ran slightly slower, but at least it worked now.

Java ReentrantReadWriteLock oddities

When I was hit with startvation of some of the update threads in BpmDj, I was a bit puzzled. After all, I did use a ReentrantReadWriteLock in fair mode. A simple profiling showed that certain transaction where substantially more heavy (a writelock taking lets’ say 10 seconds), while the databasereads would merely take 1 second.

From that I concluded that because the writelock was held longer, other threads did not have the opportunity to have a fair amount of locktime themselves. E.g: the write lock is released, the longest waiting readlock is granted: that transaction is done within a second, and the writelock is granted again. And is not released for another 10 seconds.

To test this I set up a program to create 10 reader threads and 1 writer thread. Each thread would acquire a lock, wait some time (to simulate the ‘work’ done in the locked section) and then release the lock. This would be performed in a loop for about 10 seconds. Afterwards, we could measure how much time within the lock was spend for each thread and compare that with the amount of work the thread wanted to perform.

These were the results:

Unfair Lock

The unfair lock behaved, as expected, fairly unfair. If the writer had 10 times less work than the readerthreads, its locktime would be 8 times higher. If the writer had the same amount of work, it would have 16 times more locktime and if the writer had 10 times more work, then it would be granted 50 times more locktime.

Thus:
/10.0 => *7.944541604031417
1.0 => *15.917042652441687
*10.0 => *50.5366207048361

A fair lock

When we created the read/write lock in a fair fashion, the results were more in line with what we would expect:

/10.0 => /8.810361366979999
1.0 => /1.0021791947397343
*10.0 => *9.009623837637745

That is, when the worker has 10 times less work than the readers, it has 8 times less locktime. If it has the same amount of work, it receives the same amount of locktime and if it has 10 times more work, then it is granted 9 times more locktime.

This is completely as expected, yet not something we might want, because it allows heavy tasks to block the lighter tasks.

A fair lock, prefixed with tryLock()

tryLock allows an app to check for a lock, and if it is not granted the lock to continue with something else. There are two tryLock variants. The first without parameters (tryLock()), the second with a timeout.

Trylock() screws up any scheduling that might have been in place and just barges in on anything the algorithm might be planning to do.

/10.0 => /925.0224470413133
1.0 => /99.7994977890176
*10.0 => /11.036514077119893

In this scenario, the writer thread pretty much does not get anything done. Whether it is performing 10 times less or 10 times more work, its locktime ranges from ~ /1000 to /10. This is very bad, because you might expect the tryLock to make the lock unfair, yet the results of an unfair lock (see above) are completely the opposite.

A fair lock, prefixed with tryLock(timeout)

There is a second variant of tryLock: one with a timeout; which can indeed be 0. If we apply that, we get the following results:
/10.0 /8.949334569534225
1.0 /1.0091078687281538
10.0 *9.016406673440223

which is in line with the straightforward fair lock.

In BpmDj, we used the tryLock() instead of tryLock(0), assuming that a timeOut of 0 would result in the same behavior between them.

Cache coherence and Java

You all know my good old friend Dough Lea ? A moron who instead of putting any sensible locking strategy into Java has been putting the most nonsensical thing in it ? No ? Well, let’s dig into his latest bullshit. Somewhere along the lines he decided that cache coherence is really something you want the programmer to care about.

http://www.cs.umd.edu/~pugh/java/memoryModel/DoubleCheckedLocking.html gives a demonstration of the thing most people would expect to work that does not work. Basically, if you have a synchronized write to a variable, the (non synchronized) reads from other threads can retrieve a stale state version from their cache. That means, that if you have to synchronize to write a variable, you likely have to synchronize all reads as well. If you don’t want to do that, or if you cannot do that, then you should  use the volatile keyword.

That means two things:

  1. synchronizing treemaps/hashmaps through a facade basically means also locking on read accesses. Huge performance penalty.
  2. if you have an executor array of threads that will work their ways through an ordered set of thunks to be executed, then each of these thunks has to have all their variables marked as volatile. Otherwise no cache coherence can be guaranteed and a thunk that would work on thread 1, might just not work when executed on thread 2.

It really hurts me to have to think through bullshit like this. You expect your cache to be coherent. That’s it. If it is not then why the hell am I using a virtual machine ? I could as well start programming kernels.

Android storage permissions

I’m an overall unhappy person. What else would you expect if you have to navigate that excuse for an OS, which is called Android.
This time I’m talking Android Storage Permissions.
With each new release they redefine the rules completely. So whatever BpmDj wrote in the past on your device in a _persistent_ storage will no longer be persistent in newer versions. Even the idea that, when your app requests file write permissions, it should actually have them, seems far fetched. No, before you create a directory you have to ask permission to the OS of doing so. Thereby you have to notify the user (depending on the android version) and request the permission at runtime (why do they have a manifest then anyway). Of course you are not allowed to ask for that permission in the application its main routine. No no no no no. You can only ask that permission when the first user interface element is shown. Until then you can gloriously fuck off with that nice idea of creating the necessary directories and database for your app.
 
Even worse, the user can withdraw that permission at any moment during the run of your application. That means you have to check EVERY BLOODY INSULTING file operation.
 
Now, you think ‘checking permissions’ is an easy thing to do. Also there you are completely wrong. It can only be done in an asynchronous manner. Yes. Fuck that nice function you had.
The correct answer at http://stackoverflow.com/questions/32225506/android-6m-permission-issue-create-directory-not-working gives an idea on how much retardation is exactly involved.