Catching Shoulder-Driving “Wise Guys” with Automatic Plate Detection

#.Summary (tl;dr;)

I developed a prototype app to publicly report “juega vivos” driving on the shoulder. It automatically detects the license plate number and posts it to Twitter, as shown below:

It also publishes on Twitter a status similar to the following:

Soy el auto AH2108 y me gusta manejar por el hombro. #SoyJuegaVivo #NoMeImportaNadie pic.twitter.com/n1rrw0pFmY

— Manejo Al Hombro (@ManejoAlHombro) March 19, 2017

The application was developed for completely experimental and educational purposes. If anyone wants more information about the application, you can contact me at me@demogar.com or write to me at Twitter.

#.The “juega vivo” situation

It is enough to drive in the morning along the Arraiján-Chorrera Highway or the North Corridor to notice a very particular specimen, which causes much more traffic jams than there generally is. I am referring to these little animals of creation that use the shoulder, a mandatory space for high-speed roads and that are required in case you have any damage with the car or simply to give a little space if required (like when ambulances require their passage).

It all started last Friday (March 17) when I was coming through the North Corridor. I noticed a fatal line. Twenty minutes later, the traffic disappeared: it was all due to a car that had damage to its shoulder and caused a funnel (with the aforementioned animals) that threw themselves back onto the road without caring about anyone.

Then I thought, while listening to a podcast:

“It would be great to have an application on my phone that would allow me to automatically report these people, as well as Citizen Inspector, which does not have this typified offense, but that would also detect the license plate immediately.”

That same afternoon/night I dedicated myself to researching a little more and found that it was possible to do it, so I thought it would be good to try it and make my own citizen complaints… in the end, technology is within everyone’s reach.

#.Expected flow

We would need our deployment flow to look similar to the following:

For this I decided to use a stack similar to the following (which I will explain little by little):

• OpenALPR as an ALPR/ANPR system.
• Ruby to make an integration between the web service and the ALPR.
• Sinatra (Ruby) for the web service.
• RMagick (Ruby) to draw the magenta box.
• DataMapper (Ruby) for integration with the Database.
• Twitter gem (Ruby) for integration with Twitter.
• Appcelerator (JavaScript) for the mobile application on iOS (prototype).

The intention was to develop a fully functional application in less than 4 hours on Saturday.

#.Getting to know ALPR and OpenALPR

Automatic number plate recognition (ANPR) (link) or Automatic License Plate Recognition (ALPR) is a technology that allows, through a photograph or video, the recognition of vehicle license plates. Basically they use technology like Optical Character Recognition (OCR) (link), which allows conversion to text through an image.

It is widely used for law enforcement in some countries or to simply detect fleet movements, among other uses.

Of all the alternatives on the market, I found OpenALPR, a completely free and open solution that allows you to install it on a local server.

OpenALPR is an open source Automatic License Plate Recognition library written in C++ with bindings in C#, Java, Node.js, Go, and Python. The library analyzes images and video streams to identify license plates. The output is the text representation of any license plate characters.

#.Installing OpenALPR

For testing, I decided to use two variants: one with Docker and another with Virtualbox + Ubuntu Server 16.04 LTS. For the last case, the installation was very simple and it was only necessary to execute the following line from the command line:```shell sudo apt-get update && sudo apt-get install -y openalpr openalpr-daemon openalpr-utils libopenalpr-dev

At the development level I used Docker, which was much simpler to implement. Only a few ports needed to be opened for development at Sinatra.

##### Configuring OpenALPR for Panama plates

OpenALPR has several configurations of regions (United States, Australia, European Union) and countries (United States, Mexico, Brazil, etc.), but Panama is not one of them. Panama license plates, in their construction (sizes, etc.), are very similar to the license plates of Mexico and some states of the United States.

First it was necessary to determine the different patterns of the Panama plates, of which I found the following:

pa @@#### // <- las placas nuevas, ej.: AB1234. Aplica también para Metro Bus, ej.: MB0000 pa ###### // <- las placas anteriores, ej.: 123456 pa @##### // <- las placas de Taxi, ej.: T12345 pa #@@##### // <- las placas de Taxi de Ruta Interna, ej.: 8RI12345

A file called `pa.patterns` must be created in `/usr/share/openalpr/runtime_data/postprocess` with the configuration described above.

Additionally, you must create a file called `pa.conf` and place it in `/usr/share/openalpr/runtime_data/config`. This file is the physical configuration of the board:

```conf
; 30-50, 40-60, 50-70, 60-80
char_analysis_min_pct = 0.30
char_analysis_height_range = 0.20
char_analysis_height_step_size = 0.10
char_analysis_height_num_steps = 4

segmentation_min_speckle_height_percent = 0.3
segmentation_min_box_width_px = 4
segmentation_min_charheight_percent = 0.5;
segmentation_max_segment_width_percent_vs_average = 1.35;

plate_width_mm = 304.8
plate_height_mm = 152.4

multiline = 0

char_height_mm = 70
char_width_mm = 35
char_whitespace_top_mm = 38
char_whitespace_bot_mm = 38

template_max_width_px = 120
template_max_height_px = 60

; Higher sensitivity means less lines
plateline_sensitivity_vertical = 25
plateline_sensitivity_horizontal = 45

; Regions smaller than this will be disqualified
min_plate_size_width_px = 70
min_plate_size_height_px = 35

; Results with fewer or more characters will be discarded
postprocess_min_characters = 5
postprocess_max_characters = 7

detector_file = us.xml

ocr_language = lus

; Override for postprocess letters/numbers regex.
postprocess_regex_letters = [A-Z]
postprocess_regex_numbers = [0-9]

; Whether the plate is always dark letters on light background, light letters on dark background, or both
; value can be either always, never, or auto
invert = auto

Only the sizes (304.8 x 152.4 mm, that is, 12”x6”) are salvageable from here. We use the same US detection file (detector_file = us.xml).

#.Creating an OpenALPR integration from Ruby

OpenALPR has integrations with C#, Python, Node.js, Go, and Java.

• Java: he and I don’t get along very well.
• C#: I don’t have Windows.
• Python: I like it, I have worked with it, but I have not done anything out of this world. I think it would cost me more than Saturday and that was not the intention.
• Go: I have never used it and I have no interest in learning it in a weekend.
• Node.js: I like it, but after trying it the integration didn’t work very well for me (a little chaotic for my tastes).

In the end I ended up doing my own integration with Ruby, very simple which you can see below:

class OpenAlpr
  attr_reader :output, :command

  def initialize(file)
    @file = file
    @output = []

    begin
      @output = JSON.parse(processPhoto(file))
    rescue JSON::ParserError
      @output = nil
    end
  end

  private
  def processPhoto(file)
    @command = "alpr -j -n 10 -c pa -p pa #{Shellwords.shellescape file}"
    `#{@command}`
  end
end

I basically run the command via command line, using the flag -j to return it in JSON. All of this is very well described in the OpenALPR guide.

#.Making a web service in Sinatra for integration

Let’s start from the fact that it would be much better if we could do this calculation from the phone, but I just wanted to invest part of my Saturday for this. In the end it is just a proof of concept or a prototype that I would like to improve and deploy publicly in the future. That’s why I preferred to make a web service in Sinatra that calls the integration that we already did above.

The web service would be in charge, according to the proposed flow, of:- Receive the query that is sent through the mobile application.

• Consult OpenALPR using photography.
• Analyze the result found. If it is a good result, then I would send it to the DB (so that any report can later be shared with the authorities).
• I would also post this to the Twitter account of @ManejoAlHombro.
• I would create, with RMagick, a table based on the coordinates received by OpenALPR.

To avoid this post being too long and focus on the results, we will only analyze the OpenALPR consultation process:

  # ...
  if params[:photo]
    # Get the filename
    @filename = params[:photo][:filename]

    # Write it locally
    File.open("public/photos/#{@filename}", "w") do |f|
      f.write(params[:photo][:tempfile].read)
    end

    # Analyze it with openalpr
    @search = OpenAlpr.new("public/photos/#{@filename}")
  end

  # Validate if openalpr returns values
  if @search && @search.output && @search.output['results'].size > 0
    # Get the better result
    @result = @search.output['results'].first

    # Get the coordinates for this result (the best)
    coordinates = @result['coordinates']
    first_coordinate = coordinates.first
    third_coordinate = coordinates[2]

  # ...

Again, this is a small part of the code and is only the part that consumes the integration that has already been done. The entire code is short (about 250 lines of code only).

#.Twitter integration

Additionally, I did an implementation with Twitter. A final post would look like this:

Note: This image was just a sample, this car did not drive on the shoulder and therefore I removed it from Twitter.

We noticed that the image already processed by OpenALPR and our service made in Ruby would be published, which automatically detects the plate number and we also added (with RMagick) a magenta box on the plate that it found in the place already found. A beauty since it follows the real intention of the application: to make a public citizen complaint, [just as ENA did with the people who took away the barriers to evade the toll] (http://www.prensa.com/sociedad/Ena-identidad-corredores-Norte-Sur_0_4469553138.html).

#.Mobile App Development

For the mobile application I decided to make it in Appcelerator. I have three reasons for choosing this technology:

• For me, it was necessary to project a multi-platform integration. Initially I decided to develop on iOS since it is much faster and easier to test, but the intention is to improve the project.
• As it was a weekend project, I didn’t want to venture into new technologies unknown to me (like Ionic or NativeScript). I’ve used React Native and I really like it, but I feel Appcelerator is still a little more mature. Additionally, I’m comfortable with Appcelerator and figured I wouldn’t need more than an hour to develop a working prototype (and I did).
• If I would like more a native option (like Appcelerator or NativeScript) to do direct integration and processing on the phone in the future. It’s just doing the migration through a module, something I’ve already done and it’s quite simple.

I decided that the application should have two simple screens / tabs:- A Tab to make reports, with two buttons: to do it through a new photograph or to do it through a photo from the gallery.

• A Tab to configure the endpoint or the route to the service, useful for changing environments.

The code that is responsible for using a photo from the gallery and publishing it would be the following:

function _openCamera() {
	Titanium.Media.showCamera({
		success : function(event) {
			if (event.mediaType == Ti.Media.MEDIA_TYPE_PHOTO) {
				_uploadReport(event.media);
			} else {
				alert("No es una imagen =" + event.mediaType);
			}
		},
		error : function(error) {
			var a = Titanium.UI.createAlertDialog({
				title : 'Camera'
			});
			a.setMessage('Error: ' + error.code);
			a.show();
		},
		saveToPhotoGallery : true,
		allowEditing : true,
		mediaTypes : [Ti.Media.MEDIA_TYPE_PHOTO]
	});
}

function _openGallery() {
	Ti.Media.openPhotoGallery({
		success : function(event) {
			if (event.mediaType == Ti.Media.MEDIA_TYPE_PHOTO) {
				_uploadReport(event.media);
			} else {
				alert("No es una imagen =" + event.mediaType);
			}
		},
		error : function(error) {
			var a = Titanium.UI.createAlertDialog({
				title : 'Camera'
			});
			a.setMessage('Unexpected error: ' + error.code);
			a.show();
		},
		allowEditing : true,
		allowMultiple : false,
		mediaTypes : [Ti.Media.MEDIA_TYPE_PHOTO]
	});
}

Functionally the application would look like this:

Note: This image was just a sample, this car did not drive on the shoulder and therefore I removed it from Twitter.

#.Comments and possible future improvements

I did all this development personally over the weekend. I have not done it thinking about developing a product or a mass consumption application for now. I think the product would not be production ready yet. For now, I will be using it personally and if anyone is interested, do not hesitate to write me an email or on Twitter. Since I have an iPhone that I’m not using, I think I know what I can use it for.

Possible improvements to the application would be:- Ideally, the camera would auto-detect the movement of the car or when it finds a license plate. For this it would be necessary to do the integration at the native level and integrate OpenALPR natively (totally possible).

• It would be a good idea if the mobile application worked on Android as well. At Pixmat we have a test lab, so it would be quite easy for me to migrate it.
• I would like to add other faults and failures and for the system to allow reporting, such as throwing garbage in the car, parking incorrectly, not turning on the signals, among others. There are several offenses that can already be reported by the Citizen Inspector, but others that cannot be reported, which is why I think that this application should be a public complaint.
• I should add more possible messages/hashtags to the Twitter. This is something simple. I also didn’t add (yet) the mention to @ATTTPanama.
• The app still has some problems detecting some photographs/plates. It is not determined whether the confidence percentage of the image is above 50% nor does it ask the user for feedback.
• The service additionally receives 3 other parameters that have not been configured yet: Latitude, Longitude and Comments. The intention would be for the report to also use GPS to locate the person.
• If the plate is not found, it does not give the person the option to give feedback such as manually writing the plate or simply correcting it.
• I plan at some point to publish all the source code. I feel that, since it is a job done as a hack, the result is not very optimal nor do I feel satisfied since I focused on making it work.

In the future, I plan to do some tests with an Arduino and an RPi to see if it is possible to make a similar system, but completely autonomous (maybe even try with a Dashcam). Definitely, with the traffic that these individuals sometimes cause, they could do it even at the roadblock.