Category Archives: Programming

Simple RPC With Thrift

A key aspect of building a server-based (cloud-based in today’s lingo) service is communication. The (often times remote) client needs to communicate with the server. Further, sometimes other processes on the server also need to communicate with each other. There are several ways to accomplish this, one of which is with RPC.

Many blooming programmers and hackathoners today will jump straight to “Why not just create a simple REST API using JSON serialization?”. On the surface, there are many good things about this. I’m sure you’ve heard them all:

  1. REST is simple and stateless. The semantics of a REST API are widely used and therefore easy to use once you’ve learned them. With a REST API on your server, it’s super easy to manipulate data and debug operations using tools like Postman.
  2. REST encourages readability. I’m all about readability everywhere in computing. Code should be readable, text should be readable, interfaces should be readable. It only makes sense, then, that an API should be readable as well. REST encourages this. It’s very easy to tell what GET api/users will do.
  3. REST encourages readable serializations. Since the API endpoints are readable, it’s only natural to make the response readable as well. Today, most APIs accomplish this by serialiazing data in the easy-to-read JSON format. This way, data is easy to read and easy to manipulate.

This is all well and good. Hackathoners and newcomers should not feel discouraged from using REST/JSON to create an interface to their cloud application. There’s one problem that I’m sure you’ve noticed, however.

JSON is heavy. When you’ve implemented a distributed, load-balanced, fully cached, and 100% optimized service, the largest bottleneck is transmission time from the server to the client, especially if the response object is large. In fact, on all of the teams I’ve worked on at various companies (except one), complaints about transmission time for huge serialized objects were extremely common.

Also, REST is cumbersome. I need more than two hands to count how many times I’ve coded up, from scratch, an interface layer that handles REST-style requests and serves the response in JSON. At one point I thought it a good idea to create a C# tool which actually generates a PHP REST interface when given the dataschema. Why did I have to do this? Surely, someone else has already done it!

Enter Facebook’s Thrift, which is open source and Apache licensed. For me, Thrift’s biggest features are how it solves the problems I’ve mentioned above. In order to use Thrift, you design a schema to represent both your objects and your service. The Thrift compiler then uses your schema to generate a client and server for you, meaning that you no longer have to handle the communication or serialization problems.

As a contrived example, say that I wanted to make a simple service to get my server uptime. I first design a Thrift schema:

service UptimeService {
    i32 getUptimeInDays();
}

Then, I compile the schema using Thrift

thrift --gen py uptime.thrift

I am going to use Python for my client and server, so I use the --gen py flag. Thrift has many supported languages, however.

I can then use the generated Python libraries to write my server implementation:

import UptimeService
import subprocess

from thrift.transport import TSocket
from thrift.transport import TTransport
from thrift.protocol import TBinaryProtocol
from thrift.server import TServer

class UptimeHandler:
    def getUptimeInDays():
        return int(subprocess.check_output('uptime').split(' ')[3])

if __name__ == '__main__':
    handler = UptimeHandler()
    processor = UptimeService.Processor(handler)
    transport = TSocket.TServerSocket(port=9090)
    tfactory = TTransport.TBufferedTransportFactory()
    pfactory = TBinaryProtocol.TBinaryProtocolFactory()

    server = TServer.TThreadedServer(processor, transport, tfactory, pfactory)
    server.serve()

And I also use the generated Python libraries to write my client implementation:

import UptimeService

from thrift import Thrift
from thrift.transport import TSocket
from thrift.transport import TTransport
from thrift.protocol import TBinaryProtocol

def main():
    transport = TSocket.TSocket('localhost', 9090)
    transport = TTransport.TBufferedTransport(transport)
    protocol = TBinaryProtocol.TBinaryProtocol(transport)
    client = UptimeService.Client(protocol)

    transport.open()
    print("Server uptime: {} days".format(client.getUptimeInDays()))
    
    transport.close()

if __name__ == '__main__':
    main()

Of course, this client implementation only works when run from the server itself. If I wanted to get the uptime remotely, I would just change ‘localhost’ to my domain name.

And that’s it! All of the networking details and serialization is handled by Thrift. Of course, a big portion of this post went on about how JSON is too big and too heavy. If one compares performance of various Thrift-like libraries, they will notice that Thrift is definitely not the fastest. Instead, libraries like Avro, Protobuf, and CapnProto are much faster and also more compact.

However, a major pain point for me across the years has been implementing the actual interface layer. Writing client code to summon an HTTP Request and read the response from the server gets old after a while. This is something that Thrift handles for you. As you can see from the example above, Thrift handles all serialization, deserialization, and message-passing for you. All you have to worry about is defining a schema — Thrift gives the rest for free!

Do Not Give Users Reason to Uninstall

Before I begin, I want to point out that this post is not about some evil method of user-retainment nor is it about uninstalling software on an operating systems level. This post is fitted within the scope of any single piece of security software and its owner.

During the development of my first piece of security-related software, No-CSRF, I began to notice that there were a lot of thoughts that crossed my mind which were in the same vein as “But what if the user forgets to re-enable after they disable?”. I was spending a lot of time making a Chrome extension that would make the user more safe as they browsed the World Wide Web; however, the extension was only useful if the user had it enabled.

I never did anything about this issue, however. The only attempt I made to solve the problem was to ensure that the user could re-enable the extension just as easily as they disabled it. When they disabled it, they would know that they could also re-enable it. [1]

When the first version of the extension was released, however, it was a little too strict. Sites that posed no security threat had their functionalities broken and some sites were blocked completely. In fact, one of my colleagues, who was excited to try the extension and make his browser a small bit safer, found many of these cases. After complaining that he was unable to pay an online bill due to the extension, he uninstalled it from Chrome completely and urged me to fix it.

This isn’t an issue in itself; however, it exemplifies a problematic attitude amongst users – something along the lines of “If it breaks what I need, then I don’t need it.” Even if No-CSRF was protecting users from many dangerous Cross-Site Request Forgeries, users chose to uninstall it, essentially choosing convenience over security.

Much like users uninstalled the extension, other users will take far more drastic steps in order to make things more convenient. I had another friend who, upon having port issues with a dedicated server, decided to disable his firewall. These choices, which jeopardize a user’s security, are made without much thought.

Thus, when it comes to security software, a careful balance should be sought after. Although the security-software developers would probably like their software to protect users from many different attacks, they should instead choose to balance protection and usability. A security protocol that will be used by many users is also one that allows the user to become more secure without a change in workflow.

If the developer chooses to make their security software too secure, users may uninstall, making the benefits of the security null. If the users didn’t uninstall a software with less protection, but protection nonetheless, then that is the software that should be produced.

As soon as users have a reason to uninstall, they will. Do not give them that reason.

Finding the balance between security and usability is difficult. Although it may be tempting to solve the problem by making it difficult to disable the security software as a whole, users should never be stripped of this freedom. Thus, I propose that the following tenants should be followed when making security-related software:

  1. Do not alter the difficulty of disabling/uninstalling. User freedom is just as important as security.
  2. Do not give the user an unavoidable reason to uninstall your software.
  3. Make disabling on a per-case basis easier than uninstalling.

If these three points are achieved, the user should choose to disable the security in any case where a broken workflow is not worth increased security, but should not disable the security software as a whole.

References
1 This is referring to a custom in-extension disable rather than the browser-level disable

Results of Stripping Cookies from all non-GET Cross-Origin HTTP Requests

Being inspired from a class that I took on computer security and being particularly moved by literature written on Cross-Site Request Forgery, namely a paper written by Zeller et al. [1] I decided to begin work on a Chrome extension that would locally aid in the prevention of CSRF attacks by stripping cookies from potentially dangerous requests.

The Princeton paper mentions that this can be done if the following two rules are implemented:

  1.  Allow any GET request to go through, since HTTP/1.1 dictates that GET requests should not perform action.
  2. Block (strip cookies) any non-GET request which voids the same-origin policy.

I went ahead and did just that in the development of No-CSRF, which shamelessly used a very similar extension by avlidienbrunn as a starting point.

After about a week of testing the extension through myself and some colleagues, we discovered that this method of preventing CSRF actually broke many websites, including (but not limited to) Google Drive, Facebook Messenger, and a login page for the University of California, San Diego.

The reasoning for this breakage was simple. These websites relied on functionality that sent cross-origin non-GET requests to different parts of product, each with their own sub-domain. The sub-domain, of course, is included in an origin and therefore differing sub-domains indicate different origin.

One thing to note is that the paper by Zeller et al. suggested that the stripping of cookies should not be performed if the request is valid under Adobe’s cross-domain policy. However, even when allowing the websites outlined in crossdomain.xml, functionality was still broken.

There are a few results that follow from these observations:

  1.  The local CSRF protection scheme outlined in Zeller et al. cannot be accomplished without breaking several popular websites.
  2.  In order to resolve this, the policy may be changed such that cookies are only stripped from requests that are cross-domain, where a domain is defined as everything not including the sub-domains (for instance, the domain of drive.google.com would be google.com. Thus, a non-get cross-domain request from calendar.google.com to drive.google.com would not be stripped.)
  3. On a shared system where different users control different sub-domains, therefore, it is possible for attacker.website.com to send a malicious CSRF attack to private.website.com without stripped headers.

In fact, this proposed lax “cross-domain stripping” is exactly what was implemented in avlidienbrunn’s extension before I removed it and replaced it with a stricter “cross-origin stripping” in No-CSRF.

As a result, therefore, I propose that by sending cross-origin non-GET requests, major websites are limiting the success of local CSRF prevention by forcing them to be less-strict about which requests have their cookies stripped.

References
1 Cross Site Request Forgeries: Exploitation and Prevention. Zeller, William and Felten, Edward W. Princeton University, 2008.

Rust: You probably meant to use .chars().count()

After reading a fantastic article by Ian Whitney, it came to my attention that there is some confusion regarding the “length” of a string in Rust. According to the documentationstd::string::String.len() returns the number of bytes that are in the given string. On a technical level, there is nothing confusing about this definition. However, it is widely accepted by other languages (like Java and Ruby) that the “length” of a string is the number of characters within the string.

The problem with this difference in definition is brought to light in a playpen by respeccing which shows that Rust’s std::String::String.len() function produces counter-intuitive results. Two strings with the same character count return different “lengths” because they contain a different number of bytes.

The solution to this is instead to use a String’s character iterator and count the number of elements, as std::string::String.chars().count() does.

Reddit Discussion

What Over-Mocking Revealed to Me

Recently, I took the idea of “units” in code to the extreme. After researching several programming methodologies and learning about the advantages and disadvantages of some of the methodologies, I found myself very keen of the “unit” methodology – all functions and methods are their own functional “unit”, with their own unit tests and functionality. All of these functional units should be “unit-testable”, meaning that they should not rely on a long list of other functions to be called first. From my understanding, having code like this also contributes to the Clean Code methodology, although I have not read the hailed guide.

In making code that follows the “unit” methodology, one simply assumes that all other functions and methods besides the one in question work properly. That is, there should be no unexpected bugs or kinks within them. They are summed to have complete test coverage and not have any special failure cases. Thus, when testing units of code, one can safely mock out all external functions (even those from within the same class, module, or project) and ensure that the code still follows the intended flow-of-logic.

When one attempts to write these tests, however, they will quickly notice as I did that the tests are no longer testing specific input and output, but rather that only the code-flow is being tested. That is, the only thing being ensured is the fact that expected lines are executed. Although this is a good thing because it is good to ensure that code is flowing as intended, it doesn’t actually test any specific corner cases, or that output is as expected (a big problem).

If there exists a test where all references are mocked and only code-flow is tested, it must then follow that there is another test that tests input and output, ensuring that output is as expected and that certain input generates the proper output. After all, these are the important tests that ensure that the user will not be surprised when they provide a string as a parameter to a multiply method.

However, it seems extremely tedious to think of and write two types of tests for every unit. Not only does the programmer need to think of proper ways to mock references, but he also has to think about possible corner-cases that may break his code. The aforementioned StackOverflow post [1] and long sessions of thinking allowed me to come to a conclusion.

Since the tests where references are mocked require knowledge of the actual code, these are called white-box tests, meaning that it is easy to see what goes on in the “box” – the unit of code. The tests where only input and output is tested are known as black-box tests, because the test-writer shouldn’t care what goes on in the box, only that certain input results in certain output.

Given the requirements of both white-box tests and black-box tests, it is easy to see who should be writing the tests. White-box tests should be written by the developer himself at the time of writing code. These tests ensure that there are no variables where there should have been different variables, that all necessary code executes, and that nothing is left out. The creation of these white-box tests also gets the developer to think about possible problematic inputs.

When the white-box test-assisted code is complete, the code is then given to a quality engineer, who writes the black-box tests to ensure that all inputs, no matter how wacky, generate expected results. This ensures that the end-user (whether it be other developers, clients, or simply other functions within the same module) doesn’t get stuck on any unexpected behavior. The quality engineer is the perfect person to write these tests, as he doesn’t know how the code works on a technical level, only what it is supposed to do and how it should react to certain inputs.

This makes the idea of functional “units” a bit more understandable. Someone who knows the code should write tests to ensure that the flow is as intended, and someone unaffiliated should make sure input is as expected. Of course, on a single-developer team, both jobs are for that single developer.

With that being said, white-box tests are not always necessary. If a method is simple enough, as in get_first_elem_of_array(int* arr) -> int, it doesn’t need to have a white-box tests associated with it. It is easy to see that the code should function as required. However, if a function is more complicated, a white-box test should be written.

White-box tests are something special, however. Since they are written based on the specific flow of code possessed by a functional unit, the test’s passing is entirely reliant on the code that was in-place at the time of writing the test. If the code in the function was changed, the test will fail. This may strike some as a bad thing; however, it forces the developer to design easily-testable code, even if making just a small change. After all, small changes can indeed break things, so small changes should be tested. As long as the same functionality is maintained, the black-box tests should not fail.

I am executing this newfound understanding of functional units while working on PyCFramework, and so far, it has produced very high-quality, modular, extensible code. Although writing tests takes a large chunk of time, the process of writing tests has forced me to think about the design of my code, how it could be improved, and what mistakes I may have made while coding.

References
1 https://stackoverflow.com/questions/32622040/python-unit-testing-should-other-classmethods-be-mocked/32624367?noredirect=1#comment53142597_32624367