Category: Development

Caching repetitive requests

For a client I created a php script that gets certain user profile statistics from another website and returns it as JSON to the ajax request that our own website sends.
This was the easiest way, since saving it ourselves and keeping the data up to date with all the changes would be too much work.

In order to lower the loadtime & keep the other website from being swamped with requests (every profile load) I used memcached to cache the data for 12 hours.
Caching data with memcached is fast and simple.

If you don’t have memcached yet, you can install it on ubuntu with the following command:

sudo apt-get update
sudo apt-get install php5-memcached memcached

It should instantly start a memcached process, or you can start it manually:

/etc/init.d/memcached start

the config file will be in /etc/memcached.conf or /etc/sysconfig/memcached

Here you can change the port (default it runs on 11211)

Now whenever my php script gets a request for a profile, I check if we have the data cached in memory:

mc = new Memcached();
$mc->addServer("127.0.0.1", 11211);

$result = $mc->get($cachekey);

if ($result) {
   return $result;
}

the cachekey can be anything you wan’t wich will identify the data you are looking for (in my case I used the url encoded as the key)

If no result was found, I then get the result from the remote website, and save it in my memcached server for 12 hours

$mc->set($cachekey, $stats, 43200);

This way only the first request for every profile in 12 hours will be sending a request to a remote server, decreasing the average loadtime of the pages for the visitors.

Don´t forget we can also use memcached for caching results we get from database queries on our own database, to decrease loadtime and database cpu load.

Personally I prefer varnish for full page caching solutions (I will talk about that in another post), but for smaller stuff like a hand full of objects / results memcached is a very nice option.

 

C# application for paxton door access

Paxton offers a product range including IP, wireless and battery powered access control solutions to provide reliable security for any site requirement.

We ordered one door access card reader & control unit, with it comes net2 software shipped, in wich you can manage users access cards (wich doors who can open and when, expiration date etc..)

I will use the access control for a gym, where members can open the front door with their personal (rfid) card, untill their membership expires.

There was no way to automatically re-activate the members card after they renewed their subscription online, but since the net2 software paxton develops has a net2 sdk available, I created a c# application that updates the user automatically.

In this case a simple console application would be enough, wich runs every 5 minutes, checking a directory for xml files.
If a member renews their subscription or a new member is created an xml is sent to the server where the application picks up the xml and updates / creates the user in the net2 database:

class Program
    {

        private const int REMOTE_PORT = 8025;
        private const string REMOTE_HOST = "IPADDRESS NET2 SERVER";
        private OemClient _net2Client = new OemClient(REMOTE_HOST, REMOTE_PORT);
        private static string logFile = string.Format("log/log-{0}-{1}-{2}.txt", DateTime.Now.Day.ToString(), DateTime.Now.Month.ToString(), DateTime.Now.Year.ToString());
        private string sourceDirectory = @"xml";
        private string archiveDirectory = @"archive\";
        private StreamWriter log = File.AppendText(logFile);

        static void Main(string[] args)
        {
            Program main = new Program();
            main.startUp();
        }
        
        //Check if there are xml files with user updates in the directory
        public int startUp()
        {
            IEnumerable xmlFiles = Directory.EnumerateFiles(sourceDirectory, "*.xml");
            if (xmlFiles.Count() <= 0)
            {
                Log("Nothing to do, shutting down..");

                return 1;
            }

            bool res = AuthenticateUser("OEM Client", "password");
            if(res != true)
            {
                Log("Authentication failed, exiting application");
                return 1;
            }
            Log("Authentication success");
            //Check for available xml files
            loadAndParseXml(xmlFiles);
            return 1;
        }

        private void loadAndParseXml(IEnumerable xmlFiles)
        {      
            //Check if there are files
            foreach (string file in xmlFiles)
            {
                Log(file);
                string contents = File.ReadAllText(file);
                if(parseXml(contents) == true)
                {
                    string fileName = file.Substring(sourceDirectory.Length + 1);
                    File.Move(file, archiveDirectory + fileName);
                }
            }
        }

        private bool parseXml(string xmlstr)
        {
            Log("Parse XML");
            XElement users = XElement.Parse(xmlstr);
            Log(users.Name.ToString());
            if(users.Name.ToString() == "update")
            {
                Log("This user needs an update");
                updateUsers(users);
            }
            if(users.Name.ToString() == "create")
            {
                Log("This user needs an add");
                addUsers(users);
            }

            return true;
        }

        private DateTime stringToDateTime(string date)
        {
            // Date strings are interpreted according to the current culture. 
            // If the culture is en-US, this is interpreted as "January 8, 2008",
            // but if the user's computer is fr-FR, this is interpreted as "August 1, 2008" 
            
            DateTime dt = Convert.ToDateTime(date);

            return dt;
        }

        /**
        * Update existing users in Net2
        * Field9_50 is Emailaddress (don't ask me why)
        **/
        private void updateUsers(XElement users)
        {
            var data = from item in users.Descendants("user")
                       select new
                       {
                           email = item.Element("emailaddress").Value,
                           expiration = item.Element("expirationdate").Value,
                       };

            foreach (var p in data)
            {


                //IUser interface object is returned
                IUsers userList = _net2Client.ViewUserRecords(String.Format("Field9_50 = '{0}' AND active=1", p.email));

                if (userList.UsersList().Count - 1 > 1)
                {
                    Log("Houston, we've got a problem, multiple users!! Abort Abort!");
                    Environment.Exit(0);
                }

                //Since there is always only one user, we can assume the only one in the list is the correct user
                var user = userList.UsersList()[userList.UsersList().Keys.ElementAt(userList.UsersList().Count - 1)];

                DateTime expDate;
               
                expDate = stringToDateTime(p.expiration);

                string[] customFields = new string[] { "" };

                bool blnResult = false;

                blnResult = _net2Client.UpdateUserRecord(user.UserId,
                    user.AccessLevelId,
                    user.DepartmentId,
                    user.AntiPassbackUser,
                    user.AlarmUser,
                    user.FirstName,
                    user.MiddleName,
                    user.Surname,
                    user.Telephone,
                    user.Extension,
                    user.PIN,
                    user.Picture,
                    user.ActivationDate,
                    user.Active,
                    user.Fax,
                    expDate,
                    customFields);

                Log(string.Format("Update user Status: {0}, Name: {1}, ExpirationDate: {2}", blnResult, user.FirstName, expDate));

            }
        }

       /**
        * Authenticate user
        * For some reason we can't authenticate with username password, only with userId Password..
        * So we need to get the userId and Authenticate with it..
        **/
        private bool AuthenticateUser(string userName, string password)
        {
            IOperators operators = _net2Client.GetListOfOperators();
            Dictionary<int, string> operatorsList = operators.UsersDictionary();
            foreach (int userID in operatorsList.Keys)
            {
                if (operatorsList[userID] == userName)
                {
                    Dictionary<string, int> methodList = _net2Client.AuthenticateUser(userID, password);
                    return (methodList != null);
                }
            }
            return false;
        }

        private void Log(string logMessage)
        {
            Console.WriteLine(logMessage);
            log.Write("\r\nLog Entry : ");
            log.WriteLine("{0} {1} : {2}", DateTime.Now.ToShortDateString(),
            DateTime.Now.ToLongDateString(), logMessage);
            log.WriteLine("-------------------------------");
        }
}

 

The xml files that are added in the xml directory look like this:

<update>
<user>
	<email>[email protected]</email>
	<expiration>10-08-2016</expiration>
</user>
</update>

After the file has been parsed it will be moved to the archive folder, for later reference or checking.

C# application for paxton door access

Paxton offers a product range including IP, wireless and battery powered access control solutions to provide reliable security for any site requirement.

We ordered one door access card reader & control unit, with it comes net2 software shipped, in wich you can manage users access cards (wich doors who can open and when, expiration date etc..)

I will use the access control for a gym, where members can open the front door with their personal (rfid) card, untill their membership expires.

There was no way to automatically re-activate the members card after they renewed their subscription online, but since the net2 software paxton develops has a net2 sdk available, I created a c# application that updates the user automatically.

In this case a simple console application would be enough, wich runs every 5 minutes, checking a directory for xml files.
If a member renews their subscription or a new member is created an xml is sent to the server where the application picks up the xml and updates / creates the user in the net2 database:

class Program
    {

        private const int REMOTE_PORT = 8025;
        private const string REMOTE_HOST = "IPADDRESS NET2 SERVER";
        private OemClient _net2Client = new OemClient(REMOTE_HOST, REMOTE_PORT);
        private static string logFile = string.Format("log/log-{0}-{1}-{2}.txt", DateTime.Now.Day.ToString(), DateTime.Now.Month.ToString(), DateTime.Now.Year.ToString());
        private string sourceDirectory = @"xml";
        private string archiveDirectory = @"archive\";
        private StreamWriter log = File.AppendText(logFile);

        static void Main(string[] args)
        {
            Program main = new Program();
            main.startUp();
        }
        
        //Check if there are xml files with user updates in the directory
        public int startUp()
        {
            IEnumerable xmlFiles = Directory.EnumerateFiles(sourceDirectory, "*.xml");
            if (xmlFiles.Count() <= 0)
            {
                Log("Nothing to do, shutting down..");

                return 1;
            }

            bool res = AuthenticateUser("OEM Client", "password");
            if(res != true)
            {
                Log("Authentication failed, exiting application");
                return 1;
            }
            Log("Authentication success");
            //Check for available xml files
            loadAndParseXml(xmlFiles);
            return 1;
        }

        private void loadAndParseXml(IEnumerable xmlFiles)
        {      
            //Check if there are files
            foreach (string file in xmlFiles)
            {
                Log(file);
                string contents = File.ReadAllText(file);
                if(parseXml(contents) == true)
                {
                    string fileName = file.Substring(sourceDirectory.Length + 1);
                    File.Move(file, archiveDirectory + fileName);
                }
            }
        }

        private bool parseXml(string xmlstr)
        {
            Log("Parse XML");
            XElement users = XElement.Parse(xmlstr);
            Log(users.Name.ToString());
            if(users.Name.ToString() == "update")
            {
                Log("This user needs an update");
                updateUsers(users);
            }
            if(users.Name.ToString() == "create")
            {
                Log("This user needs an add");
                addUsers(users);
            }

            return true;
        }

        private DateTime stringToDateTime(string date)
        {
            // Date strings are interpreted according to the current culture. 
            // If the culture is en-US, this is interpreted as "January 8, 2008",
            // but if the user's computer is fr-FR, this is interpreted as "August 1, 2008" 
            
            DateTime dt = Convert.ToDateTime(date);

            return dt;
        }

        /**
        * Update existing users in Net2
        * Field9_50 is Emailaddress (don't ask me why)
        **/
        private void updateUsers(XElement users)
        {
            var data = from item in users.Descendants("user")
                       select new
                       {
                           email = item.Element("emailaddress").Value,
                           expiration = item.Element("expirationdate").Value,
                       };

            foreach (var p in data)
            {


                //IUser interface object is returned
                IUsers userList = _net2Client.ViewUserRecords(String.Format("Field9_50 = '{0}' AND active=1", p.email));

                if (userList.UsersList().Count - 1 > 1)
                {
                    Log("Houston, we've got a problem, multiple users!! Abort Abort!");
                    Environment.Exit(0);
                }

                //Since there is always only one user, we can assume the only one in the list is the correct user
                var user = userList.UsersList()[userList.UsersList().Keys.ElementAt(userList.UsersList().Count - 1)];

                DateTime expDate;
               
                expDate = stringToDateTime(p.expiration);

                string[] customFields = new string[] { "" };

                bool blnResult = false;

                blnResult = _net2Client.UpdateUserRecord(user.UserId,
                    user.AccessLevelId,
                    user.DepartmentId,
                    user.AntiPassbackUser,
                    user.AlarmUser,
                    user.FirstName,
                    user.MiddleName,
                    user.Surname,
                    user.Telephone,
                    user.Extension,
                    user.PIN,
                    user.Picture,
                    user.ActivationDate,
                    user.Active,
                    user.Fax,
                    expDate,
                    customFields);

                Log(string.Format("Update user Status: {0}, Name: {1}, ExpirationDate: {2}", blnResult, user.FirstName, expDate));

            }
        }

       /**
        * Authenticate user
        * For some reason we can't authenticate with username password, only with userId Password..
        * So we need to get the userId and Authenticate with it..
        **/
        private bool AuthenticateUser(string userName, string password)
        {
            IOperators operators = _net2Client.GetListOfOperators();
            Dictionary<int, string> operatorsList = operators.UsersDictionary();
            foreach (int userID in operatorsList.Keys)
            {
                if (operatorsList[userID] == userName)
                {
                    Dictionary<string, int> methodList = _net2Client.AuthenticateUser(userID, password);
                    return (methodList != null);
                }
            }
            return false;
        }

        private void Log(string logMessage)
        {
            Console.WriteLine(logMessage);
            log.Write("\r\nLog Entry : ");
            log.WriteLine("{0} {1} : {2}", DateTime.Now.ToShortDateString(),
            DateTime.Now.ToLongDateString(), logMessage);
            log.WriteLine("-------------------------------");
        }
}

 

The xml files that are added in the xml directory look like this:

<update>
<user>
	<email>[email protected]</email>
	<expiration>10-08-2016</expiration>
</user>
</update>

After the file has been parsed it will be moved to the archive folder, for later reference or checking.

Frans Boone Store Realtime UPS Rates & Pickup points

Frans Boone wanted to offer his clients to choose between normal ups shipping rates & free shipping to ups pickup points near the client.
In order to do this I created a custom shopify app for Frans Boone, which uses the different ups api’s to get the pickup points near the customer & normal shipping rates, and return it to the shopify checkout process, where the client can choose the desired option.

 

 

SSH Key Authentication

If you are working with SSH a lot, it might be useful to generate a keypair, and setup the public key on your remote server, so you can easily login without needing to enter a password every time you connect.

Client config

A keypair consists of 2 files, by default they are located in ~/.ssh and are called:

  • id_rsa (private key, never give this to anyone, and never put in on any public location)
  • id_rsa.pub (public key, this key is meant to be configured on the server you would like to access)


Step 1. Generating a keypair with ssh-keygen:

If you don’t have a keypair yet, or want to generate a new one for this specific server, you should open terminal and enter: ssh-keygen and press enter.

ssh-keygen-1
Note: If you already have a keypair don’t use the default location, or it will replace your current private & public key

 

Enter the filename in which you would like to save the file, I’m using test for this demo purpose, press enter.

After this you will be prompted to add a passphrase, this is optional.

ssh-keygen-2

As you can see my test certificate has been generated.

Server config

In order to be able to connect to hour server without a password next time, we need to get the contents of the public key, and add it to the autorized_keys file on the server.

On your client terminal you can use cat to output the public key contents:

cat ~/.ssh/id_rsa.pub

Just copy and paste the output

Login to your server as you normally would (with your credentials) and paste the public key into ~/.ssh/authorized_keys if the file doesnt exist it will be created by calling
vim ~/.ssh/authorized_keys
or
nano ~/.ssh/authorized_keys
whatever you prefer 🙂

To doublecheck if your server config is setup correctly you can check the config file sshd_config with vim or nano (if you can’t find it use locate sshd_config)
the following lines should be present in the config file:

RSAAuthentication yes
PubkeyAuthentication yes
AuthorizedKeysFile .ssh/authorized_keys

If they are not you can add them at the bottom, and restart your sshd server with the following command:
/etc/init.d/sshd restart

 

Notes

  • If you used a different name for your public and private key (so not the default id_rsa) you will need to add an entry to your ~/.ssh/config so terminal knows which identity file to use for you server.
    an entry would look like:

    Host 127.0.0.1
    IdentityFile ~/.ssh/myidentityfile

    Note that you point to the private keyfile here.
  • If you are having trouble using your keys, make sure your private keyfile is readonly for you (chmod 600 ~/.ssh/id_rsa)

SSH Key Authentication

If you are working with SSH a lot, it might be useful to generate a keypair, and setup the public key on your remote server, so you can easily login without needing to enter a password every time you connect.

Client config

A keypair consists of 2 files, by default they are located in ~/.ssh and are called:

  • id_rsa (private key, never give this to anyone, and never put in on any public location)
  • id_rsa.pub (public key, this key is meant to be configured on the server you would like to access)


Step 1. Generating a keypair with ssh-keygen:

If you don’t have a keypair yet, or want to generate a new one for this specific server, you should open terminal and enter: ssh-keygen and press enter.

ssh-keygen-1
Note: If you already have a keypair don’t use the default location, or it will replace your current private & public key

 

Enter the filename in which you would like to save the file, I’m using test for this demo purpose, press enter.

After this you will be prompted to add a passphrase, this is optional.

ssh-keygen-2

As you can see my test certificate has been generated.

Server config

In order to be able to connect to hour server without a password next time, we need to get the contents of the public key, and add it to the autorized_keys file on the server.

On your client terminal you can use cat to output the public key contents:

cat ~/.ssh/id_rsa.pub

Just copy and paste the output

Login to your server as you normally would (with your credentials) and paste the public key into ~/.ssh/authorized_keys if the file doesnt exist it will be created by calling
vim ~/.ssh/authorized_keys
or
nano ~/.ssh/authorized_keys
whatever you prefer 🙂

To doublecheck if your server config is setup correctly you can check the config file sshd_config with vim or nano (if you can’t find it use locate sshd_config)
the following lines should be present in the config file:

RSAAuthentication yes
PubkeyAuthentication yes
AuthorizedKeysFile .ssh/authorized_keys

If they are not you can add them at the bottom, and restart your sshd server with the following command:
/etc/init.d/sshd restart

 

Notes

  • If you used a different name for your public and private key (so not the default id_rsa) you will need to add an entry to your ~/.ssh/config so terminal knows which identity file to use for you server.
    an entry would look like:

    Host 127.0.0.1
    IdentityFile ~/.ssh/myidentityfile

    Note that you point to the private keyfile here.
  • If you are having trouble using your keys, make sure your private keyfile is readonly for you (chmod 600 ~/.ssh/id_rsa)

Using version control in your development workflow

I think every developer should use some form of version control (svn, mercurial or git).
I use git for all of my projects and have used mercurial / svn in the past, it saves me a lot of headaches in the projects in the long run.

There are great ways to use git for bigger projects, like the git branching workflow:
gitflow-orig-diagram

 

But for the smaller projects this is too complicated.
For the smaller projets I would recommend just using a development branch and a release branch.

checkout the release branch on the client’s webserver (first make sure your client isn’t hosted on a shared hosting service..)
checkout the development branch on your development server, make all the necessary changes, and have the client check the changes on the development server.
If all is well merge the development branch into the release branch, and pull the changes on the release server, and your done.

If any problems arise you can check the commits for any errors, and revert them or add necessary updates and merge them again.
I always keep a wiki page on github as well for all my projects, this is always usefull if a customer calls you 6 months later.