Scaling the Minecraft Game for studying Human Behavior

I’ve created a game in Minecraft for testing human exploration in indoor environments. It’s available for playing at http://www.vaisaghvt.com/minecraft-experiment/ . Please go ahead and give it a try if you have a minecraft account and want to play Minecraft for Science . Or if you’re the competitive sort, try to get on the leaderboard.

 

It’s been more than a year since my last post on how I created the game. Since then, I’ve got some very basic results from it with students in NTU playing. However, it was surprisingly hard to get people and I had to host it online to get enough data. It took a while to get funding approved and stuff but finally I’ve managed to host my experiment on Amazon. In this post, I’ll try to describe the things I had to do since my last post to get this to work.

What I’d already done:

A short version of what I explained in my last post: I’d created an adventure of sorts in Minecraft where players get spawned at one location, follow instructions to explore a three storey building and then complete a few tasks. On disconnecting from the server (manually), all the data of the actions of the player is written to a MySQL server. I analyse this data in the hope of finding a cure for cancer. And this is what I had to do for that:

Step 1 : Shutting down the server automatically

I can’t believe how long it took me to do this, but it ended up being quite simple with my modified statistician plugin simply sending a shutdown command to the server once the player completed the last task.

Step 2: Starting on request

Next, I used Python’s brilliant twisted framework to write a simple server program that would listen for connections and start a tekkit server when such a request was received. To test things out I created a simple client side script also that simply sent the username. The database would be queried for existing attempts and the unique id consisting of the username and attempt number was created to store it in the database.

TO start and stop the server I used this handy little script: http://pastebin.com/Lgs4r5f8 .

Step 3: Hosting the server and scripts on AWS

Now that I had this simple set up working I got an amazon EC2 machine and a MySQL database and copied all my server files and scripts there. Instructions on how to do this are easily available on googling and quite straightforward to follow.

If anyone’s trying to copy a Tekkit server from a Windows to a linux machine, remember that you have to download the Tekkit server on the machine separately and just copy the world files (as opposed to trying to copy the whole folder there).

Step 4: Final Touches

Now that I had most of this working, I added a new page and with some simple python and javascript and a bit of effort in beautifying things, I was able to send requests to play from my website to the EC2 server and play.

I also made the server a white listed server with the white list being created on a person signing up. Finally, I put a time out on the server that would shut down the spawned tekkit server after an hour so that the experiment didn’t stop simply because someone started the server and didn’t play.

The end results of all this are there for all to see at http://www.vaisaghvt.com/minecraft-experiment/ .

 

Final Comments

I didn’t go to as much detail as I wanted simply because I don’t have the time to write in detail. I’ll try to share all the scripts and stuff I wrote on my GitHub page once I get time to clean it up a bit. In the mean time, if anyone’e curious to know more about any part of the process, do ask.

Also, if you play minecraft and have a minecraft account or know anyone who does, please go to http://www.vaisaghvt.com/minecraft-experiment/ and play minecraft for science 🙂

Each Tekkit server with all the plugins I have require about 3G of memory, so it’s expensive to get more than the one EC2 instance that I have now and making a proper queueing system that emails people when the server is available will take a lot of effort so as of now, only one player can play at a time and someone else who requests while the one player is playing will simply have to wait and check back later.

I’m working on a paper summarizing the results from my analysis. Shall share that also once I’m done with it.

 

Making a game to study how humans explore

 I made a first person exploration game in minecraft which I’m using to study how human beings gain and store spatial knowledge. This post outlines the processes involved in making and getting data for analysis from a minecraft server.

A little bit of background first: my PhD thesis is on modeling human behavior in simulations of crowds. More specifically it’s about modeling crowds that are evacuating buildings during fires, terrorist attacks or something similar. As a part of my thesis, I’ve been planning to develop a model of spatial knowledge and exploratory behavior of humans; this could be invaluable in structuring buildings in a way so that evacuation is quick, efficient and relatively painless. In order to get some data on how humans explore, we came up with the idea of making a game which could subsequently be analysed to determine something about human behavior (It’s not us who came up with the idea to use games to analyse stuff. There has been an interesting Ted talk on it). It’s been a really interesting process.

Making a game from scratch

I started off trying to make an actual game from scratch using the Unity Game Engine. Pretty soon, I realized that even though I could make a game of sorts in unity without much difficulty, I would be spending an inordinate amount of time designing the environment, the characters, the game play, etc. At the suggestion of my prof I decided to check Minecraft out as an alternative.

Resorting to Minecraft

For those who don’t know, in Minecraft, the player takes on an avatar that can destroy or create blocks, forming buildings, structures, artwork and even entire cities on multi-player servers or single player worlds across multiple game modes. It’s a simple and fun enough game. In the original version, in the day time you gather resources to make shelters and weapons and at night monsters come out to destroy you and you have to protect yourself. With the game having no specific goals and the player having the freedom to do a lot of things, despite the extremely simple graphics, it’s turned out to be insanely popular. Users range from gamers to hobbyists to teachers trying to teach computers and programming in an interesting way to kids.

Getting instructions to kick off

Creating a world

Why did it make sense for me? Well, in minecraft, I could use the creative mode to get access to unlimited resources and make anything I wanted without having to bother with monsters. There were also some tools that made this part even easier. The worlds thus created could be exported and used any time. There were a lot of people who’d made some amazing stuff ranging from the Taj Mahal, to Hogwarts to scaled models of cities. Some of these amazing creations can be found at http://www.minecraftworldmap.com/ and the like. So now I could create an environment and modify it easily to structure my experiments as needed. After some research and testing I zeroed in on this amazing little palace.

I could edit it however I wanted and basically save a snapshot of the world I create by copying the “world” folder. At any point to start from that world, I could just replace the world folder in the currently running minecraft server instance and voila! I had a building/environment for my game. All I needed to do was make a game out of it.

Making a game in minecraft

The other great thing about Minecraft besides custom worlds are the mods.  Modifications of the Minecraft code, called mods, add a variety of gameplay changes, ranging from new blocks, new items, new mobs to entire arrays of mechanisms to craft. And there were hundreds, probably thousands of these online. There were also some particular flavors of minecraft like bukkit and tekkit that came preloaded with a lot of amazing mods. So I got tekkit and I had some pretty neat wireless transmitters and receivers and ready made NAND gates (You can create your own NANDgates in vanilla minecraft but this is incredibly hard and very messy). How would I use all these tools though? Again with inspiration from some podcasts and youtube videos and after discarding some ideas like making puzzle games and treasure hunts, I finally found the perfect idea for a game that would satisfy my research requirements and could be made by a beginner like me in minecraft without much difficulty. And I called it …. (drum-roll) ….

The Great Escape

Well, I do need to come up with a better name for it; but the basic idea was that a player would be made to explore the building doing some basic tasks. This would be the exploration phase. Following this, he would be given some tasks to do which would require him to use some of this knowledge he’s gained from his exploration. More specifically, the player had to find 11 prisons distributed around the palace and then fulfill a couple of tasks to escape. But how would I go about making this?

Prisons with glass windows and an iron door. Using the lever opens the door and sends a wireless signal.

Making the prisons

Well, first, there was the relatively simple task of making the prisons. I manually made them using creative mode and some standard blocks, placed a villager avatar inside it and made an iron door with a lever next to it. In minecraft this meant that the iron door could only be opened with the lever. So after closing of certain parts of the huge palace and placing the prisons in such a way that finding them would require the player to explore at least 90% of the palace, I had to design the tasks to do after exploration. This was going to be much more difficult since I had to somehow hide these tasks and reveal them only when all the prisoners were freed.

Hidden tasks

This is where the NAND gates came in hand. Using a bunch of wireless transmitters connected to each lever I managed to create a simple circuit that would open secret walls only when all 11 prisons were opened or some tasks completed. My basic game was ready.

Making it a game

I had my environment and my game, there was still one big issue though: this was still minecraft. So the players could still go around destroy things I’d made and mess up the environment. This is when I turned to a different kind of mod: plugins. Minecraft is basically just java based and some people had been nice enough to create APIs of sorts and tutorials and stuff for it which could basically be used to modify the game to our liking. Plugins were basically java jar files that could would put on the server which modified the game in some way. With a bit of searching through the huge library of plugins and some help from some forums I found and modified some existing mods so that the players could now only interact with doors and levers.

Funky what simple logic gates can do… That’s the logic of my game.

Now all I had to do was copy my world file and the mod that controlled the interacting permissions to my server and start it. I also installed a plugin that ensured that players who joined spawned at a particular location. After that if I joined the world I would start at the same location and be forced to play the game in the way it was meant to be played. But how would this help in my research?

Getting data

Here again, I found an existing plugin for a minecraft server called Statistician made by ChaseHQ. It’s a pretty neat tool that enables the minecraft server administrator to analyse statistics about what resources where gathered by each player and how much they ran. It wasn’t too hard to add a few lines of code to track the players locations at each time step as well.

Making it more interesting

Now that I had access to some code that could track the user, it wasn’t too hard for me to make the game a little more game-like with some interactive messages. I first made a timer that tracked how long the user had been playing and gave him/her a countdown of 45 minutes with a 1 minute time step.  I then changed this to include some preparation time for people who hadn’t played first person games before.

What was more interesting was adding some more custom events to the tracker. I added an event that would be triggered when a prison was opened, storing the lever ID (based on location) and the time it was opened in the database. This gave an idea of which prison and task was completed at what time. Now that I was tracking when prisons were opened I also gave warnings to players when they closed prison doors inadvertently, which was a common mistake that players made.

The actual setup

So basically, what I have is the game server running the world I created with all the prisoners, levers, prisons and stuff. A player could join the server and be spawned at the “starting location” for the game. Here he/she would follow instructions given from the server and complete the game. Once complete, he/she would quit the game. On the player disconnecting, all the data till the point of the last task being completed needed to be written into a database. After this I needed to restart the server with the same world in it’s pristine state again so that someone else could play. I made a script that could automate most of this. The player had to disconnect and the server had to be stopped manually. After this a script could be run that prompted for a name to store the latest data under. Once stored in the database, the world was copied into the server folder and the game restarted.

Pros and Cons of using Minecraft to do this experiment

Advantages:

• Can relatively painlessly create new environments.
• Could go on to study the effects of communication in escape by allowing more players to join the server and play at the same time. (I currently allow only one at a time for reasons I’ll discuss in a bit.)
• The awesome mod community: There are countless existing mods and more importantly for me, it’s just simple java so I could add my own mods quite painlessly.
• The game structure on the server is quite simple: easy to add mods and change worlds. And script the whole process.

Limitations :

Well, they’re mostly limitations of how I did it either due to lack of time/ interest or in some cases a lack of knowledge and not limitations of minecraft per se:

• Only one person can play at a time. Unless I get more servers, any interactions made by one person to the world would be seen by others since they were playing in the same world.
• I only had the school server to run on which is behind a corporate firewall and is thus quite slow to connect to remotely. So I had to be in the University network to play reasonably smoothly.
• I still haven’t figured out a way to kill the server when a player disconnects and restart a new instance of the server with the pristine environment so that someone else can connect.

There are other limitations as well with regard to what conclusions can be drawn from a game reliably. But since I haven’t discussed the game or the structure of the experiment itself here. I don’t think it’s relevant to discuss this at this point.

There were certain minor issues I encountered along the way, not the least of which was making the game playable by people who had never played games before and for people who had difficulty understanding english properly. I still don’t have enough data, but some of the things do look interesting. Shall post about it later when I have something definitive.

A link to the modified plugin for anyone interested : github repo.

A Sublime Text plugin for the careless

So after close to a year of procrastinating, I finally made the code that I had written about back in June last year into a plugin for sublime text 2. The plugin basically checks for a couple of mistakes that are commonly made in latex by me and hopefully other people who forget to title case titles and put spaces after punctuations. On running the command, the plugin highlights each mistake and suggests a replacement which can be accepted or rejected by the user with a single key press. IF you’re actually reading this post, do feel free to check it out at https://github.com/vaisaghvt/CheckTypos .

The process of making it into a plugin was quite straightforward since my original script was in python. After some help from the excellent NetTuts tutorial and some help from some of the existing plugins and obviously the API I had the plugin working. About 10 lines of JSON later, I had an option in the tools menu, a key mapping for mac, linux and windows and a command in the command palette.  Seriously, I’ve no idea why I waited for so long.

 

p.s. It’s now available in package control as CheckTypos plugin. Love how that things works.

Some thoughts

I had initially planned to write just about my PhD related stuff on this blog. However, the sheer ridiculousness of the stuff I see on facebook and in the news these days has lead me to write this post.  It’s actually about a few random, somewhat unrelated news articles and posts I’ve been seeing that scare me actually. I’ll start off from the most recent and work backwards from it.

Rehabilitate Pakistani Hindus: BJP (http://bit.ly/PQWIWR)

This came on the Times of India on Aug 13th. I’m all for helping refugees in whatever we can. What bothered me more was the emphasis on hindus and the comments below the article, many of which had more than a dozen likes/ recommendations. Most of the commentors actually want India to provide dual citizenship to Hindus all over the world like Israel does for the jews and send all muslims away to Pakistan or Bangladesh “as it was meant to be”. Or if they aren’t going to go away, they want muslims to sit silently like a good minority should. In case this needs explanation, this scares me because what we have here is a bunch of people educated in english and capable of reading newspapers who have this image that not only is India a hindu nation, but muslims are foreigners and not a part of this country. We have the third largest Muslim population in the world. You know what scares me more? That this isn’t just a bunch of random fools with access to a computer. The second largest national party in India practically thinks along these lines. Which brings me to the next thing that irritates me.

The NaMo brigade

For those not in the know, that refers to the people who want Narendra Modi to be the next PM of India. Now, let me get this clear, I am extremely impressed with the work that Modi has done in developing Gujarat and I do believe he is a much more capable administrator that Manmohan Singh and (God Forbid.. ) Rahul Gandhi. And India might well be better of with him in power than another term for this astoundingly inept Government of ours. What scares me is not him per se; but his followers. For the following reasons:

• Modi’s recently been trying to paint a picture of being more secular and inclusive of muslims. And his followers and others in his party find this reason enough to stop believing in him. Get this, we have a large chunk of people who are pissed of that a prime ministerial candidate is trying to care (or at least show that he cares) about another community!
• I don’t claim to know whether he orchestrated the Godhra riots and that isn’t my major issue. There have been cartoons going around about how people seem to ignore all the good that he has done because of one incident. What scares me about this cartoon is the thought behind it. His supporters aren’t saying that he didn’t do it. Had this been the case, I would definitely not have been so scared.  They are actually saying that even if he did do it, look at all the good he did.  This scares me because if he did do it, it shows what he is capable of. And by showing that they are capable of forgiving such an act, his supporters reveal a very scary side of India. *
• Somewhat related to the previous point. There is a large chunk of people apparently who don’t even believe the “even if” scenario, they believe that he did do it and are happy about it! Apparently there used to be riots every few months or years but since the Godhra riots the Muslims have been scared and quiet and there have been no riots. This is apparently good. So much for Tagore’s dream of living in a country “Where the mind is without fear and the head is held high”

Talking about living without fear and holding your head high brings me to my next topic for this post: the way women are treated in India.

The condition of women in India has been highlighted a lot over the last few months by different parties: Aamir Khan’s excellent Satyamev Jayate brought up some of the gravest issues facing women in India; Tehelka brought to light the alarming apathy of the Indian Police to women in India; and there have been various articles across different blogs, newspapers, websites about the problems facing women in India (including a scathing article in the Guardian and a report that ranked India the worst country to be a woman in). But despite all this, it’s when I see some of the things my friends on facebook post and reactions to it that I start wondering how screwed up things really are. Some examples:

Case 1 : Pictures of girls standing in line to buy alcohol: There were two of these and dozens of likes and comments mostly along the lines of how our so-called “culture” is degrading. Most of these comments, shares and posts are by guys who themselves drink themselves stupid every chance they get.

Case 2: (1.2 maybe considering how related it is to the previous one) Pictures of some girls smoking with a detailed post on how “westernisation” has now destroyed “Indian culture”: I think smoking is a disgusting habit. But girls have as much right to smoke as a guy. Another thing about this that I hate is this idea of “western culture” that India is obsessed with. The less said about it the better. All I have to say is that, I think India’s a better place because there are girls who’re not scared of smoking because they are girls.

Case 3: Gehna Vashisht and the bikini controversy: If you didn’t know, this female did a photo shoot wearing an Indian flag coloured bikini. An idiotic thing to do and done solely to raise a controversy and get some publicity I think. Some “social activists” reacted by throwing stones at her and getting her beaten up. As wrong as what these hooligans did was, that is also not what I have an issue with. Following this incident, posts appeared on facebook from some of my “friends” about what good, patriotic citizens these hooligans were beating up someone who had insulted the nations honour. What’s more, you had to like it if you wanted to beat her with a shoe, share if you want her beaten with broomstick and comment if you want her shot. I’d rather not go into how many likes, comments and shares there were. Why was this even worth sharing?? There are so many other issues in society to get angry about and this is what works these people up? Brilliant.

These are people who went to the same schools as me. In my naive mind they weren’t supposed to be the problem.  The problem was supposed to be in some other corner of India, in uneducated villages and in the older, more traditional generation. When such opinions and thoughts seem to be so widespread in the youth of today, it just hurts.
* For those who want to bring up the Assam riots and Sikh riots and Rajiv Gandhi’s comments as points against the Congress. Two things: Firstly, the assam riots were a case of incompetence and not malice. I can live with incompetence. I’m scared of malice. Secondly, I’m not a congress supporter either. They were very wrong in doing what they did but at least they finally apologised for it.

Anyway, it’s going to be the 65th anniversary of India’s independence soon and anyone who’s actually been reading this post till now might find it interesting to read the dream with which India awoke to life and freedom 50 years back: Nehru’s speech.

The NTU option for undergraduate studies

Every year, mostly around this time of year, I get a lot of friends, relatives and complete strangers asking me about NTU and considering it as an option for further studies. Depending on how pissed off I am at NTU at any given time, my opinion changes but the gist of it still remains the same. It depends on what you’re comparing it against and what you want from it.

This post is basically my thoughts on the choice I made 6 years back and hopefully it’ll help someone or the other in their dilemna. Ever since the day I decided to take the SIA NOL scholarship and come to NTU and study computer engineering instead of aerospace/ chemical engineering at one of the IITs, I’ve spent an unhealthy amount of time justifying to myself and to others why I made a reasonably good decision. When I was at the point of having to choose, one of the things that helped me the most was an alumni’s blog post about how he made the decision to come here. I can’t remember his name right now, but I remember finding out about a year or so back that he’s currently doing his graduate studies at NUS. Anyway, the point is I can’t find his blog now and I hardly find anything written by people who decided to come to NTU instead of going to one of the IITs. So here’s an attempt at helping a lost soul considering his options.

Firstly, some disclaimers. I’ve only studied in NTU and a semester at the University of Waterloo in Canada. I have not really stepped into campus much in any other college (except NUS a few times and IIT-M for some competition when I was in 12th Grade). Secondly, the NTU experience is completely different for different people. I’m guessing this is the case in most universities. But the sheer size of the student population, the variety of extra curricular activities on offer and (when comparing to Indian Universities) the amount of freedom you have ensures that what I say is very likely to be completely different from a lot of other students.

On the positive side, moving to a foreign country is always an eye opening experience and while most of the students are from Asia, there is still an incredible variety in the backgrounds of the different students here. While it’s not necessary and quite common that most of your friends will be from your same country/ background, the international exposure does make quite a difference in your outlook. More importantly, Singapore is, regardless of the criticism that some people put at it, an amazing city. And, there is a lot one can learn from the way things are done here. It’s as safe as a city can be and for those wanting to travel it’s also very well connected to a lot of amazing places.

NTU also has some brilliant facilities for pursuing your different interests. I joined the NTU Tae Kwon Doe team and Red Cross during my first year, attended a few (kinda intimidating) sessions of the competitive programmers club, organised some cool events and some not so cool ones. I even got paid during my second year for doing research in NTU’s initiative to try and encourage research amongst students. Whatever be your interest, there’ll be something or the other for you, especially if you’re willing to take the effort.

Moving on to the more pertinent topic of academics, the aforementioned choices and freedom still applies. While most courses have a few core subjects that you have to take over the years there is an incredible variety of subjects that you can take which are offered by the different schools ranging from Maths and the Basic Sciences, to Arts, Humanities or Business or even things like Physical Education (I recently discovered that there are summer courses on soccer and other sports). You can take minors in most of these fields to go along with your major in Engineering (or the other way around). Pretty soon, NTU’s going to have a medical school too. While these options aren’t really a big deal to someone in NUS/ most major universities abroad, I think it is quite different from most Indian Universities ( I could be wrong of course). Another great thing about NTU is that, NTU has enough money and prestige to attract some of the most brilliant scientists and personalities from all over the world as visiting faculty or for talks that happen regularly on campus.

This brings me to the topic which is generally considered the most important: the quality of the teaching. I’m not sure how good or bad things are in other universities, but compared to my experience at the University of Waterloo and listening to online lectures from various US universities, the quality of the teaching and (more importantly) the learning culture in NTU in general is extremely disappointing. I believe this is a combination of a lot of factors.

I can count on one hand the number of classes I’ve actually enjoyed sitting through at NTU. And there wasn’t a single class that inspired me to want to learn more about computers or computer science or computer engineering. This is not entirely the lecturers fault. For one, the culture of asking questions is entirely absent here. And as a result, even when the lecturers do try desperately to encourage their audience there is absolutely no reaction. Secondly, there are no attendance requirements and all lectures (at least till third year) used to be recorded. So most students think they can make up for it even if they don’t attend classes. So it’s not uncommon to see very empty classrooms. However, what I hated most about NTU was how easy I found it. Most tutorials and assignments were not challenging or open ended or even remotely interesting. After all the awesomely interesting problems that I came across during IIT preps, these tutorials and assignments were very boring and rarely required thinking outside the box. In fact, some of them didn’t even require thinking. The reason why I found my semester at Waterloo to be so interesting (and that semester was the key reason for re igniting my interest in computer science enough to do a PhD) was the assignments which were generally quite difficult.

Many graduates from computer engineering have never used UNIX, had never programmed in anything other than Java, have never done any Computer systems course and have a very very basic knowledge of algorithms.

However, there are three reasons why my rant should be taken with a pinch of salt. Firstly, I think I would have ranted about the quality of teaching in many other colleges too. Secondly, there are a lot of changes coming about and things might already be a lot better. Classrooms are being made smaller, course loads are being reduced with more importance being given to assignments and quizzes and some effort is also being put in training TAs. Lastly, this is the case in the school of computer engineering in the courses I took. I did attend some really good and interesting economics and maths classes. And I’m sure teaching styles and standards can be very different in the other schools.

To end on a more positive note, in a lot of cases, especially for Indian students, what is more important than what you learn in college is your career prospects and stuff. And in this regard, NTU and Singapore in general does not disappoint. The compulsary Industrial Attachment program and the general culture of taking summer internships ensures that you are prepared when you graduate. In spite of the recession causing some difficulties, I think all graduates do find decent jobs after graduating. Even if you don’t end up getting an awesome pay (and there are quite a few who do get good jobs), your quality of living in Singapore will be a lot better than most other parts of the world while the cost of living is actually not so bad. Most students go into the financial sector since that pays best, however there are engineering firms too for those so inclined. For the business minded, the Government and Schools (the minor in entrepreneurship is extremely popular) here do a lot to encourage and support startups. Despite not having their engineering departments in Singapore, Facebook, Microsoft, Amazon, Google and others do come to campus every year and do recruit quite a few students. So, as a stepping stone for a future career and a comfortable life you could definitely do a lot worse than NTU.

There are a lot of other factors that can affect someone’s choice of NTU for undergrad: On the negative side, the bond that is there for students on scholarship is something that needs to be considered very seriously. On the positive side, Singapore’s quite close to India and if you book at the right time you can be to India and back for less than 10000 INR. There are a lot of other personal factors too. In my case, even though there are a lot of things I dislike about NTU, given the circumstances, I still think I made the right decision in coming here.

Regex + Python to clean up my writing

After shifting to using latex (with sublime text) for writing, one of the things I’ve found rather irritating is correcting the silly mistakes I keep making. These are things like accidentally putting two spaces adjacent to each other, repeating phrases and forgetting to capitalize letters in the right places. Word used to make things easier with it’s spell check. There is a dictionary in sublime text but it works only for spelling mistakes and those aren’t always the problem. Checking the PDF and going through them looking for mistakes was obviously quite irritating.

Initial Solution

It was around this time I made a list of standard regexes that I could search for and replace using sublime text’s in built search. I put these in my sublime text latex cheatsheet for easy access. I’d just copy these from the cheat sheet and paste into the search bar and fix each error as I saw it. Obviously quite time consuming. I intended on automating this with some sort of script but just never got around to doing it. With all the other checks I had to do like whether the text appeared in a comment or an equation block or something. I had no clue how to do this in a simple bash script.

Python To the Rescue

It was at this time I was reading about someone using regexes in Python and I realised this would be an interesting way to improve my limited python skills and do something useful. And I set about making a python script that checks for the common mistakes I make (the regexes in my cheatsheet) and makes the appropriate suggestions for replacements and updates the file.

Adding new regex patterns and ways in which it has to be replaced is as simple as writing a simple function and adding a line to the list of patterns to be tested. I still need to do some basic testing on it and add more patterns but I’ve put the code up on GitHub already (link) and would be extremely grateful to anyone who checks it and gives any suggestions. I will update the Readme file and comment my code very soon (seriously.. i will ).

Finally, using the idea of functions being first class members in python for the first time was super interesting and super useful. Gives a hint of some of the biggest limitations of java; and a brilliant rant by Steve Yegge on java and functional programming: http://steve-yegge.blogspot.sg/2006/03/execution-in-kingdom-of-nouns.html.

My own little space on the world wide web

It’s been a very long time since I’ve been planning to get one done, but I’ve finally got a personal website/ academic profile up. It was almost half a year or a year back that my supervisor told me to create a website for myself and it’s somewhat essential for PhD students and stuff. But NTU wasn’t giving me any domain space and though I could get some server space on the PDCC servers for creating a team website and page for myself, that never materialized either because everyone was too busy. So I created some random thing on sites.google.com. Which looked pathetic and I couldn’t really figure out how to customize it much and most importantly I wanted my own domain.

This is not the first time I’ve made a website. But the last time I created a personal web page I was in 11th Standard (Junior College) and it was created using Front page and I think about 5 people in total saw it. I created it using geocities, I wonder if it’s still up. Anyway, after that I’ve started a few blogs now and then and I went through the initial process of trying WordPress and then Joomla for creating the team website. But for my own website I wanted to create it form scratch using just html and css nad not using pre specified templates.

So the first step was learning how to use CSS and brushing up on the basic HTML I knew and checking out html 5 (not that I needed it, but might as well see what the big deal was I thought). Anyway, with the help of some tutorials and templates from Lynda.com, I was able to create the website quite easily.

Next I had to get it up on the interweb. I’d heard from some friends that Amazon Web Services has some free tier of storage if I use less than a certain amount of space. And being amazon, I’m guessing the service is reliable. Anyway, I set up an account. Even though I had to submit my credit card details for it, I won’t be charged for the first year unless I add more than 5 GB of data to the storage provided. Also, they’ve allowed me to set an alarm if the charge ever goes above 0$.  Anyway, they have excellent instructions on how to use it. However, I don’t really think it is even necessary. It’s that easy. Just remember, DO NOT delete buckets randomly. It takes a while to have that name available again.

Now that I had my data stored there, the website was accessible but the link was a very long and weird one and it basically made the geocities/ google sites domain name look better. So I started searching for a name registrar. The first link and recommended article I got to was LifeHacker’s recommendation. Since I wanted the cheapest one out there, I went for namecheap.com. But a quick search online, showed that internet.bs provided domain registration for about a dollar less and people seemed to be reasonably satisfied by it.  So I went ahead and bought vaisaghvt.com for about 8.99USD a year.

The process was easy enough. But being  newbie I had no clue how to link that url to my aws bucket. I initially tried simple url forwarding. While this worked (and incredibly quickly at that ) it ended up showing that long useless URL as the URL in the address bar which I found quite irritating and inconvenient. Luckily I found this very helpful blogpost, which told me exactly how to solve this problem. Just remember, to name your bucket properly. In my case since I wanted it to be linked to www.vaisaghvt.com, this is the name that I gave to my bucket. While they kept telling me that this might take hours or a day to happen. In my case, I could see my website at that name in less than an hour.

I know it’s not that big a deal to create a website. But can’t help the childish glee in having my own little place in this interweb thingie.

Please do give it a look : vaisaghvt.com

Running simulations and analysing data

My first post in a long time. This is more as a journal entry for me to look back at when I need to.

My PhD project mostly involves running simulations of hundreds of people evacuating from a building and then analysing the simulation in various ways. While the MASON framework in Java helps a lot in the implementation of the model itself, something just as intersting and some thing that in the end feels a lot cooler is running all those simulations and getting data and analysing them.

Step 1: Running multiple Simulations

MASON allows you to run simulations in two major ways : Either using the GUI in which you get to see how the simulation is going. This mode is very useful and essential when creating and debugging the model. However when it comes to actually running simulations and gathering data for analysis, this is quite obviously impractical. This is when the console mode comes in handy. In the console mode, you run several replications of the required simulation with the required seed. Initially I used the handy in built function to do this. I also needed to store the simulation specific settings in some place. Initially I did this using constants in various classes, which I changed to storing all the constants in one class which was a lot more convenient to change and finally I resorted to a much more practical xml file which can easily be read from using JaxB in java. Though I think I might change to an sql based implementation soon. Anyway, the point is, I am able to run my simulation using it’s jar file and an xml file with all the parameters that are used for the simulation.

Step 2: Storing data:

The next step in this process is collecting data from these simulations. As a way to get started I stored my initial files as simple text files in csv format which I analysed in excel. Pretty soon, this became extremely impractical because of the amount of data I had to store. So I changed to storing in binary format and created  parser which would convert generated binary files to text files. I could have used some of java’s inbuilt analysis tools like some provided by the apache framework, but I was quite lazy, and I was working with someone who wanted the text files so that he could analyse it in Matlab, so I resorted to a binary file with a parser to convert to text.

However, despite the organised file hierarchy and names, this was still very difficult to analyse and keep organised and it was still very huge. Also there were a lot of complications when I were writing from multiple runs, experiments, etc. So I switched to what I should actually have done: a relational database. I set up a mysql server instance on my lab computer and wrote all the required database to the file at the end of each run of the simulation.

Step 3: Analysing the data :

Excel being boring, I shall not go into the details of how I did it initially. So once I got the data in MySQL, I needed some tool to analyse it. That’s when my prof recommended using mathplotlib in python. I’ve used python before to create a simple script to clean up references in a text file however, I’ve hardly used it for anything else even though I liked the language a lot. So I decided to give it a try. Interestingly enough I had a lot of trouble finding a free library for mysql. Though once I finally found, mySQLdb, the process of querying and analysing the data and getting some neat graphs took hardly a few lines of code. So now once I had the data, i could simply run the python script and get all the charts I needed.

Step 4: The power of the cloud

A single run of my simulation can take up to 5 minutes. For 100 replications of  under 6 different settings (this is what I needed for the particular run at that time) this would take about 3000 minutes or 50 hours or just over 2 days. While not bad, I needed my computer and I worked at the parallel and distributed computing center so it would have been a waste to not make use of all that computing power at our disposal. So I got myself an account on the cluster and created a simple shell script that would run the simulation with the fixed settings. Eventually I extended this so that it would read parameters from a separate text file, modify the xml file appropriately, and then run the simulation the required number of times and finally, at the end of the run I would be send an email. Here is the code for this first script:

 
#!/bin/bash
# runSimulations = runsSimulations from inputs in file 1

opath=$PATH
PATH=/bin:/usr/bin

case $# in
  0|1) echo 'Usage runSimulations settingsFile xmlFile' 1>&2; exit 1
esac

awk -v xmlFile=$2 '
BEGIN {totalCount=1
startingPoint[1]=1}
{
  model[NR] = $1
  startingPoint[NR+1] = startingPoint[NR]+NF-1
  for(i=2;i<=NF;i++){
    completeValuesList[totalCount] = $i
    totalCount++
  }
}
END {
  startingPoint[NR+1] = totalCount-1
  for (j=0; j<=NR; j++){
    indices[j] = 0
  }

  while(indices[0]!=1){
    timeNeeded=0

    for(j=1;j<=NR;j++){
      value[j] = completeValuesList[startingPoint[j]+indices[j]]
      command = "overwrite " xmlFile " xmlParser " model[j] " " value[j] " " xmlFile
      # print command
      system(command)
    }
    testCommand = "grep FilePath " xmlFile;
    testCommand |getline filePathLine
    close(testCommand)
    seed = 1
    javaCommand = "java -cp dist/CrowdSimulation.jar app.RVOModel -repeat 100 -time 100 -seed " seed
    # print javaCommand
    system(javaCommand)
    for(j=NR;j>=1;j--){
       if(startingPoint[j]+indices[j]==startingPoint[j+1]){
          indices[j]=0
          indices[j-1]++
       }else {
          if(j==NR){
             indices[j]++
          }
       }
    }
 }
}' $1
echo $1 $2 "run complete"|mail -s "Run Complete" vaisaghvt@gmail.com

For anyone with a little experience in shell scripting, this might seem like crap, so if are bored enough to go through this and you know some shell scripting, please do give me any suggestions that you have. That was the code for my first project. In my second project, I’ve changed my approach to having a separate class for each experiment. And also initially, I manually did the work of connecting to each cluster and initializing the job. Now, I’ve automated this too. So I specify the experiment and settings to be run and the script dispatches the jobs to the specified set of clusters and as above, I get emailed at the end when the data is available.

#!/bin/bash
opath=$PATH
PATH=/bin:/usr/bin

case $# in
  0) echo 'Usage runExperiment classToBeRun' 1>&2; exit 1
esac
program=$1

for cluster in "c0-0 0" "c0-1 20" "c0-2 40" "c0-3 60" "c0-4 80" "c0-5 100"
   do
      set -- $cluster
      ssh $1 "nohup ./runCommunication.sh $program $2 2> 2_$2.log 1> 2_$2_1.log < /dev/null &"
      echo "assigned to $1"
   done

SSHing to a remote client and running the command in nohup were the two most difficult parts of this. Nohup lets you run the process even after disconnecting from the machine. The & at the end makes the process run in the background so that you can disconnect and connect to the next machine or do other things. The output is redirected to log files so that I can keep a track of what is happening and finally, something that I took a long time to figure out, you should set input to be received from /dev/null, otherwise you will not be able to disconnect from that particular remote machine.

#!/bin/bash
opath=$PATH
PATH=/bin:/usr/bin

case $# in
  0|1) echo 'Usage runSimulation classFile parameter' 1>&2; exit 1
esac
java -cp IBEVAC.jar $1 $2

echo "$1 $2 run complete"|mail -s "Run Complete" vaisaghvt@gmail.com

There’s still a lot more automation I can and plan to do. But as of now, I’m in a state where I can run simulations quite easilly and I won’t be changing things much for some time. Next stop, getting a proper gitflow happening with Netbeans or eclipse.

Sublime Text Latex Cheat Sheet

(Scroll to bottom for the pdf cheat sheets) 

The quest for efficiency

I’ve been looking around a good editor for latex ever since I started using latex.

When I had this problem when I initially started using Latex during my FYP, I settled for WinEdt which seemed to be the most popular one out there. Since I had to finish my FYP report quickly (I had about a week to setup and learn latex and write my report in it), I didn’t bother experimenting with different editors. The problem for me with latex on windows was that it took me a really long time to set everything up. And for some reason, I couldn’t typeset my tex files into pdf within winedt. I’ve heard this is possible now. I’m not sure if I just never figured it out or the feature wasn’t available in earlier versions.

In any case, once I started PhD and more specifically when I was about to start my confirmation report, there were whole new complications. I had a mac at home and a pc in the lab. I would need to work on my report at both places. I could use Winedt or some such on Windows and TexShop or TexWorks on OSX. But I really didn’t like either of these apps and I wanted to learn to properly use one powerful text editor so that I could work better. So I did a check online and the most fanatical text editor ppl were ppl who used Vim or Emacs. I started using Emacs . Not entirely sure why, but I guess it’s because I found a windows version of emacs and because I heard of Aquamacs for the mac. Within a day or two, I realised that this would be pointless for me. For one, I like using my trackpad and mouse and I like having a proper GUI regardless of how powerful the app can be without it. Moving the cursor in the app was itself a pain, so I didn’t really want to do much more. I was sure there must be some modern text editor that I can use to do all the stuff I need without having to waste so much of my time trying to learn Emacs. I mean, there had to be something better than notepad++ and Textwrangler that was also free.

Sublime Text 2

That’s about when I came across Sublime Text (http://www.sublimetext.com/). It’s powerful, beautiful, has a lot of customizable key bindings, plugin development opportunities (using Python) and most usefully at that point of time it was cross platform. And yep, it was completely free. You could choose to donate to the developer (Jon Skinner) to a licensed version. However, the unlicensed version did all that the licensed version did but with an annoying pop up that comes up at every 10th save or something.

I use it generally for C/C++, python, xml, html and latex. You can use it for all sorts of things. The app also has a console which can be used for doing all sorts of things. For one, when working with python you can actually compile your files and run them from Sublime Text itself. Please take a look at the website for their features. There seems to be a lot of activity on the plugin and package development front over the last few months.

To start installing and discovering packages. First install Sublime Text 2  (http://www.sublimetext.com/2) and then install package control (http://wbond.net/sublime_packages/package_control). And voila! Installing and discovering packages becomes as easy as taking candy from a baby.

Now to start using latex on sublime, you’ll want to install the latexTools package (http://tekonomist.wordpress.com/about/). It provides all sorts of features to you including forward and inverse search, wrapping text in environments and commands, auto completion of references, etc. Combined with the shortcut keys provided by Sublime Text and Papers2’s Manuscripts I have now an Amazing setup for writing my papers. Things have never been easier.

But to actually make full use of these, a cheat sheet is absolutely essential. So I’ve created a pdf of key bindings in sublime and latex tools that’ll be most useful for anyone using sublime for latex.

Cheat sheet:

Windows:  sublimelatexsheet – Windows (p.s. any and all suggestions are welcome)

Mac : sublimelatexsheet -Mac

Tex files for cheat sheet: Tex source files for cheat sheet

*Edit : added some regular expressions that I use for checking to the cheatsheet.

*Edit: Changed for new plugin settings
Update: Installing and updating latextools is a lot easier now. And the plugin is a lot more awesome now. I don’t even need Paper’s manuscripts (which is available now in windows also). Most of this article is old but the latex tools plugin is still updated as of Oct. 24 2012.

*Edit: For updated regexes and a plugin to use these in sublime text please refer to my newer post : https://vaisaghvt.wordpress.com/2013/04/17/a-sublime-text-plugin-for-the-careless/

Why you might not want to use Word in the first place…

In my last post I talked about how to use MS Word properly for reports.  However, for any report more than than a few pages long with a few references, equations and pictures MS Word starts getting cumbersome to use. Placing pictures in a huge report is generally very irritating. It never stays where you want it and sometimes it just disappears from the page altogether.In fact, the difficulty in putting in figures is the best reason to not use Word.  Refering to specific chapters or sections or figures can be difficult. Restructuring the report messes everything. And in a report of a reasonable size , this is bound to happen.

Much more importantly, when you write a report in Word, you’re constantly thinking of how things should look and where they should come, the fonts to use and the styles to use. If you don’t, you’re most likely going to have to restructure a lot after you’ve written everything and this will lead to the same problems I said earlier.  Also, while most people do have office, or at least some software like Pages or Writer that converts the docx files to a “usable” format, a “usable” format almost never looks like it does in Word also there are (increasingly) a lot of people who don’t use Word.

LaTeX is an alternative I’d suggest. Firstly, Latex isn’t for everyone. It’s plain irritating to use if you’re new to it and just writing something small.  But for anything like a thesis or your final year report or your dissertation, LaTeX is the way to go.

In latex, you simply write the text in a generic text file called a .tex file and typeset it using Latex to get a PDF file. The tex file can be opened anywhere on any system since it’s simply a text file. You don’t care about the content and placement, you simply write your content and put in references for images. Latex takes care of putting images properly and getting your references to figures correct. Writing equations is made easy and the equations look beautiful. In fact your whole report looks beautiful and very very professional. Page numbers, tables of content, bibliographies, etc. can be generated with a single command and their formats and styles can be changed easily. Most conferences specify their prefered style and to change styles is as simple as writing a single command or copying a few lines from the specified format.

I won’t go into details here since there are a lot of resources online for this. But I would just like to ask everyone with at least some time to spare to try it. Most resources on setting up latex are just a google or bing search away. Do contact me if you want any help/advice on starting using latex or if you want a few templates to get you started (I’ll upload them anyway, when I get a little time). A few resources to get you started:

http://www.osnews.com/story/10766 : An excellent , slightly long article on why you might want to use latex and also provides a hello world and other basic stuff in latex…

http://www.stdout.org/~winston/latex/latexsheet.pdf : a cheat sheet for latex necessary for anyone using latex regularly.

http://en.wikibooks.org/wiki/LaTeX : A comprehensive and detailed wiki about everything you might want to know in latex.

http://ftp.itb.ac.id/pub/CTAN/info/lshort/english/lshort.pdf : A not so short introduction to latex.

http://www.ats.ucla.edu/stat/latex/ : A bunch of links for latex.