Picscout ‘s development process at a glimpse

The role of software engineers

    At Picscout, each SE takes full responsibility on a task. This is achieved by using the following guidelines:

• User stories (can also be refer to as tasks) are written and described in advanced, either by product owner or R&D team, and inserted into a queue.
• When a Software Engineer (SE) is available, they pull a US from the queue and start to work on it.
• The SE should read the US and verify they understand it completely. For that they may turn to the person who wrote the US, other group members, group leader, operation team or anyone who can help them understand the US.
• If the US is not well defined in such way the SE can't start working on it, they should raise a flag and talk to the person who wrote it. In this case, the US may return to the queue and it will be reviewed again.
• US should not take more than 5 days. If there's a US which we think will take longer, it should be divided into smaller User Stories. It is up to the SE to decide and divide the US.
• Once they started to work on a US, they should set a due date for it (no longer than 5 days as discussed previously). This due date helps planning the team schedule and set milestones.
• A SE should handle each US end to end. Understanding, designing, implementing, testing and deploying are part of the task.
• When a US is finished, it should be flagged as delivered or done. If the US involved code changes, it should be flagged as ready for 'Code Review'

Code quality

Maintaining our code in high quality is one of our main goals. In order to achieve this goal we use a variety of processes and tools:

Coding skills and analysis

We use code analysis tools such as FxCop and Sonar to give better insight into the code quality at both the developer’s and the project’s levels. At the same time, we put a big emphasis on the human factor:

• Code reviews are done on a daily basis
• Code reads are done once a week- A team member will show code that he/she has written to a group from R&D and the group will discuss the code (design, architecture, implementation considerations etc…)
• Clean code lessons and educational meetings are done every 2-3 weeks
• Special events such as hackatons and code retreats are done every 6 weeks 

Tesing

We do not have a QA team. We don’t believe we need one. Why?

Because we believe our software engineers should write code that works – that’s as simple as that.

hat seems like a very naïve idea of how software development can work in the real world. So how can we achieve this goal in practice?

When we develop new code we also develop some layers of testing for it:

• Unit tests (Using NUnit)
• Integration tests (Using NUnit)\
• Acceptance tests (using Selenium, Specflow and NUnit )

we do have a small team of automation verification engineers that help us manage and thicken the Acceptance testing layer Some manual testing are sometimes needed of course, and our software engineers do manual testing whenever required.

To support the development of maintenance of our products we also require a very efficient process of CI – this was discussed in our previous post.

The CI process at PicScout

 We've been using Jenkins as our build server for some time now, but recent switch from TFS to Git allowed us, among other things, to implement a more sophisticated approach to Continuous Integration process.

We've already had all CI principles covered before - single branch everyone was working and committing to on a (almost) daily basis, automated self-testing builds etc. But if you look at the definition of CI in Wikipedia, it says that in CI "no errors can arise without developers noticing them and correcting them immediately". Unfortunately this was not our case.

But first of all, why errors resulting in a broken build are such an issue ? Because it impacts entire team, anyone "getting latest" is likely to encounter issues at some stage because of the errors introduced by someone else and it may take a lot of time to realize that it's not your fault after all. And a check-in on a broken build makes things even worse. In addition, you are not guaranteed to have a stable version for deployment.

Why did it happen for us ?

A little bit of background to start with - we have one job (build in Jenkins) per solution, and we have dependencies set up between jobs to reflect dependencies between projects in separate solutions. Whenever job is run successfully, it will trigger its dependent jobs, so a check-in may actually result in multiple Jenkins jobs running in chain one after another. The whole process used to take more than 30 minutes, with developers waiting all this time for possible error notifications whereas the source was being dirty since the check-in. Moreover, any other check-in during this time frame somewhere along this chain resulted in jobs near the end of chain aggregating additional changes from TFS. Consequently, failure notification emails were sent to a group of developers and they were in no hurry to take responsibility for something that was not necessarily their fault.

What do we have now ?

First of all, we put a lot of effort to minimize jobs execution time and right now the longest chain of jobs completes in well under 10 minutes. But after migration to Git a major change was in our new strategy to tackle "broken builds". Each developer now works on a local branch, pulls from the central repository master branch but pushes back to his/her personal branch. Git Plugin allows Jenkins to merge master branch into this personal branch, run all necessary jobs on it and merge it back to master on success. In case of a failure, the broken branch is ignored, master branch remains untouched.  Any pushes made by other developers at the same time are run separately on different branches and don't affect each other. Feedback on success/failure of the build is sent only to the developer who triggered it, so no more lame excuses.

  

This concept is not new for CI servers, there is a "Gated Check-in" in TFS or "Delayed Commit" in TeamCity. What makes our approach a bit different is that the process is automated - no need to specify which build definition you want to use for your changes to be built, tested and pushed back to master. We have incorporated logic in Git post-receive hook that inspects the changes made by the developer, identifies and then triggers corresponding job in Jenkins. Another advantage compared to the 2 methods mentioned above is that the branch with broken build can be easily accessed by other developers for review or assistance. In fact, with this approach it can even be more productive to allow a broken build than to try to prevent it all the time.

That's about it on how we do CI these days.

How to split an array in C#?

Every now and then, we need to perform mundane operations that are very simple but don't have a built-in function in the language. So we write some ad-hoc code, maybe even copy something from StackOverflow and are done with it. What we sometimes fail to notice, however, is the affect this has on performance.

This entry will focus on splitting an array, but this is relevant for other operations as well.

The time it takes to perform an operation is only important if the code is in a time-critical section and/or is performed a large amount of times. If this is not the case, whatever implementation you choose will probably be OK.

Having said that, let's look at some of the ways to split an array. We'll use a medium sized byte array for this purpose (256), splitting it to a short array (16 bytes) and the rest.

Assuming you are using .Net 3.5+, the most natural way to go about this is to use LINQ:

private static void SplitArayUsingLinq(byte[] data)
{
         byte[] first = data.Take(16).ToArray();
         byte[] second = data.Skip(16).ToArray();
}

As you can see, the method is very elegant, only 1 line of code required to create each part. In addition, it seems like using Take and Skip on a small number of runs can't cause a performance issue.

However, running the above code a 1,000,000 times takes around 10 seconds, which is a considerable amount of time.

Let's compare the LINQ version to a good old for loop:

private static void SplitUsingForLoop(byte[] data)
{
 byte[] first = new byte[16];
 for (int i = 0; i < first.Length; i++)
 {
  first[i] = data[i];
 }
 byte[] second = new byte[data.Length - first.Length];
 for (int i = 0; i < second.Length; i++)
 {
  second[i] = data[i + first.Length];
 }
}

This yields a run time of less than 2 seconds – more than x5 improvement! The looping method seems to be much more efficient. Let's try to improve this some more.

If we do our Googling right, we find that copying arrays actually is a library function – Array.Copy. Let's test this:

private static void SplitArrayUsingArrayCopy(byte[] data)
{
         byte[] first = new byte[16];
         Array.Copy(data, first, first.Length);
         byte[] second = new byte[data.Length - first.Length];
         Array.Copy(data, first.Length, second, 0, second.Length);
}

We get a result of 250ms – another x8 improvement, a total of x40 compared to the LINQ version!

Digging even dipper and only in the case of byte[], we can use another method called Buffer.BlockCopy that actually performs a low level byte copy:

private static void SplitArrayUsingBlockCopy(byte[] data)
{
         byte[] first = new byte[16];
         Buffer.BlockCopy(data, 0, first, 0, first.Length);
         byte[] second = new byte[data.Length - first.Length];
         Buffer.BlockCopy(data, first.Length, second, 0, second.Length);
}

Now the results are 180ms, which is yet another improvement, albeit not as dramatic as the previous ones.

In conclusion:

Method	Time for 1,000,000 splits (s)	Improvement factor
LINQ	10.2	-
for loop	1.93	x5
Array.Copy	0.25	x40
Buffer.BlockCopy	0.18	x56

Kids, don't trust LINQ blindly for what really matters J

GetHashCode() Is it that important ?

Introduction

The Idea for this blog entry came up after a pair code review session.
I’ve noticed the following method (from class that holds picture coordinates):

public override int GetHashCode()
{
    return X.GetHashCode() + Y.GetHashCode(); // Algo. 1
}

At first glance it looked like a fair implementation

I know a few rules of thumb for solid GetHashCode implementation:
•  Hash algorithm must be deterministic (for given input, output must be the same).
•  Equal objects must have the same HashCode.
•  Objects with the same HashCode aren’t necessarily Equals.

This entry overview implementation of GetHashCode.

Preliminary testing

Why it isn’t a solid implementation?
Well let’s test that, the testing will reveal the answers.
The test define image 1024 X 1024 in dimensions, 1M pixels in total.
Our class represents a single pixel on the image,
and override the GetHashCode() method.
Now, let’s calculate the Hash Code for each and every pixel on the image surface.
Values will be compared and summarize according to colliding Hash Code values.

The number of collisions per pixel will indicate the pixel color:
White for collisions free pixels and darker colors correlated
to the “popularity” (see legend) of the Hash Code.

This produces the resulting collisions map for the 1st algorithm result:

 

The results are even worse than expected!
This is mainly due to the fact that .Net implementation of int.GetHashCode()
returns the int value itself!
So,
Pixel(X=50, Y=55).GetHashCode() == 105
Pixel(X=55, Y=50).GetHashCode() == 105
Pixel(X=54, Y=51).GetHashCode() == 105
And so on and on...

Suggested alternative algorithms

Let’s try a slightly better solution:

public override int GetHashCode()
{
    return X.ToString().GetHashCode() 
              + Y.ToString().GetHashCode(); // Algo. 2
}

You can expect using Xor operator between the X and Y
will produce considerably better results:

public override int GetHashCode()
{
    return X.ToString().GetHashCode() 
              ^ Y.ToString().GetHashCode(); // Algo. 3
}

Common solution is to use "Mersenne prime"

public override int GetHashCode()
{
    return X + Y * 31; // Algo 4.
}

Assuming that images dimensions are restricted to 64K (X or Y)
we can create "Perfect Hash"

public override int GetHashCode()
{
    return (X << 16) + Y; // Algo.5
}

Testing output

2nd algorithm result:

3rd algorithm result (Although it looks better, it's even a bit worse):

4th algorithm result (good distribution across the field,

pretty impressive for such simple algorithm):

No use to show the 5th algorithm result, the image is perfect white,

no collisions at all!

Collision effect !

Why the collisions are so important?
Mainly, due to one big reason: Performance !

Let’s try to measure the time it takes to:
Insert / lookup in dictionary (Dictionary<XyPoint, int>) and calculate Hash Code.

This table illustrates the time it takes (1 million POCO items)
for our Algorithms (tested on Intel E8400 machine):

Time in ms.	Algo. 1	Algo. 2	Algo. 3	Algo. 4	Algo. 5
HashCode Calculation	34	280	283	30	31
Dictionary Insert	19,300	2,587	2,758	547	72
Dictionary Lookup	18,614	2,468	2,692	577	98
Total time	37,948	5,335	5,733	1,154	201

Conclusion

Collisions cause huge performance impact.
Even a good Hash algorithm can suffer from a bad implementation
due to incompatibility or misuse.
Although Algo. 1 was one of the fastest Hash calculations,
the overall performance compared to Algo. 5 were about 190 times slower!
This is due to the reason that colliding values are chained,
this requires doing additional search to find the value in the chain.
That’s incredible, small and grey line of code can degrade (or boost) the performance.

Probably next time you’ll override the GetHashCode(),
you’ll spend a little bit more time to find the better solution.

I hope this entry shed some light on the subject and emphasize its importance.

Apprenticeship Program

All professionals around the world need to be trained and software engineers aren't an exception.

Hence, we announce a unique program (and for sure the first in Israel) we are proud to kick off this week: a Software Development Apprenticeship Program.

PicSocut will hire and train apprentices; We will focus (but not limit) on clean code, reading/writing code, clean architecture, BDD, TDD, simple and business oriented designs, tools and best practices. In nutshell, all what you need to become a highly competent software engineer who cares and proud about his profession.

We are looking for couple of candidates to begin the program!

If you feel it's you, please feel free to send us your resume at jobs@picscout.com

Good Luck!

  

Building Lightweight Products

Here is a short talk about how we build lightweight products at PicScout (in Hebrew).
Unfortunately, the video has focused on the speaker, instead of on the slides... so ping us if you wish to receive the slides.

Consumption of Large Json objects from the IIS

In my previous post I talked about binary serialization of Large Objects. Today, I‘m going to talk about the consumption of such objects from the IIS.

In this case our recommendation is: always stream objects, don't create intermediate strings or similar, because you will find yourself with "Out of memory etc..." exceptions.

First, let’s update our client so it will ask for “gzip” stream:

public class SpecialisedWebClient :  WebClient
{
      protected override WebRequest GetWebRequest(Uri address)
      {
           HttpWebRequest request = base.GetWebRequest(address) as HttpWebRequest;
           request.AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip;
           request.Timeout = 90 * 60 * 1000;
           return request ;
      }
}

Next, an Asp .NET application usually has a controller which returns some JsonResult. We introduced the new class LargeJsonResult which is returned instead:

public class LargeJsonResult : JsonResult
{
       public override void ExecuteResult(ControllerContext context)
       {
            context.HttpContext.Response.ContentType = "application/json";

            if(ReturnCompressedStream())
            {
                 context.HttpContext.Response.AppendHeader("Content-encoding", "gzip");

                 using (GZipStream gZipStream = new GZipStream(response.OutputStream, CompressionMode.Compress))
                 {
                      SerializeResponse(gZipStream, Data); 
                 }
            }
            else
            {
                 SerializeResponse(gZipStream, Data); 
            }
       }

       private bool ReturnCompressedStream(ControllerContext context)
       {
            string acceptEncoding = context.HttpContext.Request.Headers["Accept-Encoding"];
            if (!string.IsNullOrEmpty(acceptEncoding) && acceptEncoding.ToLowerInvariant().Contains("gzip"))
            {                
               return true;
            }

            return false;
      }

      private static void SerializeResponse(Stream stream, object data)
      {
           using (StreamWriter streamWriter = new StreamWriter(stream))
           using (JsonWriter writer = new JsonTextWriter(streamWriter))
           {
                JsonSerializer serializer = new JsonSerializer();

                streamWriter.AutoFlush = true;
                serializer.Serialize(writer, data);
           }
      }
}

Here we use the Newtonsoft Json Serializer. As you can see, the Json object is streamed to the Client, so the last thing what we need to do is to update the Clients' code for the consumption of this object:

using (SpecialisedWebClient client = new SpecialisedWebClient())
{
      client.Headers.Add("Content-Type: application/json");

      using (Stream stream = client.OpenRead(serviceUri))   
      using (StreamReader reader = new StreamReader(stream, System.Text.Encoding.UTF8))
      using (JsonReader jreader = new JsonTextReader(reader))
      {
           JsonSerializer js = new JsonSerializer();    

           return js.Deserialize<JObject>(jreader);
      }
}

That's it. Now you can transfer GBs of Json data over the wire.

Note: In my previous post, I talked about large objects serialization and our custom implementation of ISerializable. Apparently, when the Json re-presentation of this object is streamed from the IIS using Newtonsoft serializer it calls to ISerializable method and instead of Json the binary stuff is displayed. In order to disable this behavior we need to add the attribute [JsonObjectAttribute] on top of the object.