Author Archives: brandonio21

unittest.mock in Python

Python is a language that allows for fast iteration. In fact, it is this fast
iteration that makes prototyping
one of its primary use cases. Because of this fast iteration, I don’t feel as
guilty about re-writing programs in Python as I would feel rewriting programs
in C or Rust.

This also makes Python both easy and satisfying to learn. It is easy in the sense
that any new concepts can be quickly incorporated into existing projects. It is
satisfying in the same sense, where newly learned concepts can be quickly put
into practice, producing immediate results.

Thus, the Adventures with Python project is a perfect way to reinforce my
passion for coding and learn more about the language itself.

Mocking

Testing applications is something that is fascinating to me. To have a test suite
that exposes stupid mistakes and easy-to-fix errors is like a golden ticket to
a successful application. Constructing good unit tests has always been challenging
for me, however. For instance, in one of my recent applications I had code to
test whether or not file-reading function succeeded.The module looked a little something like:

import filemanager # A 3rd-party library
def read(filePath):
 return filemanager.read(filePath)[5:]

The test looked a little something like:

import util.filereader
import unittest
class TestFilereader(unittest.TestCase):
 def test_read(self):
 with open('filename', 'w+') as openFile:
   openFile.write('fakercontents')
self.assertEquals(filereader.read('filename'), 'contents')

Although there is nothing immediately wrong with this code, it produces some
issues. For instance, the test relies on file creation, meaning that if it is
run with insufficient privileges, it will fail. Further, it has the nasty side
effect of messing with the file system.

That’s when I discovered unittest.mock,
Python’s way of solving this exact issue. With Mock, I could essentially mock
and alter anything pertaining to my module that could make my tests better.
If I wanted to improve the above test using mock, for instance, I could do the
following:

import util.filereader
import unittest
import mock
class TestFilereader(unittest.TestCase):
 @mock.patch('filereader.filemanager.read')
 def test_read(self, mocked_filemanager_read):
 mocked_filemanager_read.return_value = 'fakercontents'
 self.assertEquals(filereader.read('filename'), contents')
 mocked_filemanager_read.assert_called_with('filename')

This makes the test much simpler and much more robust. The test is no longer
relying on the creation of files or on the reliance of Python file-creation
libraries. It also doesn’t have any nasty side effects.

Most of my work with Mocks has been done in a professional setting, so I cannot
share any real-world code; however, I plan to use Mocks in the testing
of PyCFramework, so be on the
lookout for that if you’re looking for real-world applications.

An Update on Python

Leave a reply

Upon looking at the title of this post, you may be under the impression that
I am starting to learn python from scratch. Although that is most definitely
what the title implies, I am doing something a little bit different.

For the past 6 months I have been using python. I first started with a hackathon
project Recipr, where I implemented a server backend using Flask. This gave me the confidence I needed to start using python for other projects.

By its own standards, python is a language that allows you to quickly prototype
ideas. With that being said, it’s perfect for homework assignments where all
that matters is the final answer. (I don’t think many students are very proud
of the actual code that is submitted along with these assignments). Thus,
I also used python to implement several machine learning algorithms for a course
I was taking at the time.

Since then, I wrote a working replacement for my Programming Competition
framework, PCFramework, in python. I uncreatively called it PyCFramework.

I also used python for a fairly-large professional project during an internship
I held this summer.

With that being said, it is obvious that I have a decent amount of experience
with python. However, there is a clearly defined difference between knowing
a language and really knowing a language. To simply know a language is to
code in that language. To really know a language is to understand its principles,
design paradigms, and hidden features that make the language great.

Over the course of the next few months, I intend to really know and understand
python. Here is what to expect:

Progression guided by Intermediate Python
Refactoring and recreation of old projects, such as machine learning algorithms
Creation of new projects, such as a new version of PyCFramework
Discussion of the topics via this blog

We Are All Humans

Leave a reply

As a citizen of the United States, it is no secret that my privacy has been in
constant violation for the entirety of my life. Despite claims of
All-American freedom, the only freedom I seem to have regarding my privacy is
the freedom to have no say into whether my privacy is breached.

The privacy issue began to be brought into the common light when Edward Snowden
revealed the ongoings of the NSA and its gratuitous spying on the American
public. This was a few years ago. Recently, however, it was revealed that AT&T
and the NSA share an agreement wherein AT&T will voluntarily aid the NSA in the
collection of data. Indeed, in several court cases against AT&T regarding their
data sharing, the United States provided AT&T with immunity, leaving them
unpunishable for their sharing of customer information.

It is easy to quickly agree that all AT&T customers should simply switch service
providers, taking their money to companies who do not have agreements with the
NSA, and thereby solving the problem. However, it isn’t that simple. If AT&T has
an agreement with the NSA that supposedly grants them immunity in court, not
many other companies would refuse such a good deal.

This raises a very big question: What gives AT&T the power to share other
peoples’ information with the government. What’s more: What gives the NSA
the right to demand this information?

What is privacy?

A longwithstanding fact is that people need some sort of privacy. Whether it
be their rooms, the bathroom, or simply privacy of mind, humans need some
place to be able to think and ponder without being watched.

Most Americans believe that they have a certain amount of privacy. For instance,
after work most Americans return to their private homes, where only a select few
individuals are allowed access. Even then, humans tend to wish for “private
time”, wherein they sit alone and contemplate things far beyond their own
personal experience.

Not only this, however. Many Americans yearn for a privacy of record. That is,
there are some things that simply need to be unexposed to everyone else. Phone
call transcripts, chat history, internet history, medical records, letters,
photos, and passwords are all examples of this. This is something that most
Americans believe they have as well. For instance, it is often considered too
Orwellian to admit that every word you say is being watched. This consideration
is testament to how much privacy Americans think that they have.

This, of course, is all understood. When it comes to the question of sharing
this privacy, many Americans will respond with something along the line of
“I have nothing to hide”! These same people, however, are reluctant in sharing
all of their personal information, passwords, and private chat transcripts
with random individuals on the street. I thought you had nothing to hide!

Governments are People

It has long been said that the United States is a nation run “by the people,
for the people”. In its literal sense, this statement simply means that the
government, the body that supposedly runs the entire country, is made up of
people. Of course, the statement was meant to be more of a democratic ideal,
showing that the same people that must follow laws have the power to change
laws.

Looking at the literal interpretation makes sense, though. After all, the
government is just a large collection of people who do various things. In
addition, the government is a small subset of the entire human population. Thus,
the government is simply a collection of humans. Just like the group it governs.

I’m not saying that the concept of a government is wrong, however. It should be
fairly obvious that without a central source of rule, things go haywire. There
is no standardization of units, no agreed upon rules, and no way to make sure
that the gullible are not abused by the persuading. This governmental group
of humans, however, are not special in any way. They, as humans, do not have
any advantages over the people they govern. They were not blessed by a god,
they are not more intelligent, they are not more knowledgeable, and they are
not more important. With seven billion people on the planet, how is it possible
that one human is more important than another? It isn’t.

With that being said, why would a certain group of humans be allowed to violate
the privacy of another group of humans? The answer is simple: They shouldn’t.
After all, privacy is something that all humans need, and for one group of
humans to deny this from another group of humans simply isn’t right.

The Current Situation

Currently, it is the case that this exact situation is occurring. A group of
humans, the United States Government, is violating the privacy of the humans
that it governs. It is not doing so in the simplistic sense that it is listening
for certain keywords to prevent terrorist attacks, but rather it is doing so in
the sense that it is storing everything. With its NSA program, the government is
storing your location of every moment of the day (via cellphone GPS), your
calls, your texts, every photo you take, every minute you spend using water,
every thing you purchase, and every person you interact with. With this, it
is creating a searchable profile for you, wherein any government official can
search for you, by name, and find out everything about you, including the things
you do in “private”.

Of course, the humans that are being governed don’t have much of a choice. By
simply living in the United States, they signed an invisible contract saying
that another group of humans, the government, had the right to invade every
aspect of their private lives.

This, of course, should not be happening. Following the ideals of all men
being created equal, one group of humans should not be able to invade the
privacy and the lives of another group of humans.

We are all humans. We are all the same. No one should have privilege over others
in the magnitude that the United States government has privilege over its
citizens.

Twin Peaks

Leave a reply

How can anyone possibly believe
that they are more than a speck?

Houses and buildings down below
with movement far from sight.

This is how it will be
when we fade away.
Life’s beauty is temporary.

When I sit up here, where the land seems to touch the sky, and the insignificance of life is made clear, I think about my goals. I only have so much time to experience and enjoy human life as it currently exists. With that being said, I think that the concept of “settling down” is reserved for a later time – for when my children need a stable place to grow and mature. Until then, while I have no obligations elsewhere, I plan to travel the world and absorb the human experience from many differnet viewpoints. I plan to work hard doing something that will help the advancement of humanity – not just on this planet, but hopefully on others as well. Only when I have satisfactory stability will I “settle down”, hopefully raising my children without at the same time struggling to meet standards set by society, as my parents struggled to do.

The last years of my life have been spent catching up with what humanity has accomplished in the past: The advancements made in mathematics, the sciences, and the arts. It is important to know how we made it to this point, so that we may humbly advance to the next. Every day, each individual aids in the advancement of the human species, each in their own way.

Migrating my VPS to Linode and Sentora

Leave a reply

Background

If you’ve been following my recent ventures at all, you know that not too long ago I decided to move my virtual private server to DigitalOcean. I also decided to run CentOS7 and use Centos Web Panel (CWP) to allow anyone using my VPS to have a nice graphical interface to manage their services. I have not taken time to formally review either of these services, but I did experience some issues with Centos Web Panel. Some of the key issues I had with the product are:

Obfuscated Code. I know that some people like to obfuscate their code in order to make the program uneditable by other parties; however, if I am unable to look at the code of my web-panel to see exactly what it’s doing, I am worried that the web-panel is doing something that it doesn’t want me to see. On top of this, obfuscated code makes it very, very difficult to quickly edit files if I want to make changes.
Support is not good. The CWP forums, which should be a central source of information for all who choose to adopt CWP into their own servers, is riddled with posts describing common issues. Almost all of the threads contain an administrator response prompting the original poster to file a ticket. This creates a forum that is essentially useless since no actual problem solving takes place there. On top of this, when an administrator does take the time to answer a question, the answer is usually not completely answering the question. To be fair, for a free product, support is something that doesn’t need to be offered; however, a community-driven forum is important, and CWP is not running theirs properly.
Things break. More often than not, users will make changes that unintentionally break things, and these breakages are reported as “random occurrances” by the very users who caused them. Maybe I am just a really poor user; however, during my time with CWP, things would definitely seem to randomly break. People on the forums seemed to agree, too, as all issues that I had were also reported by other users. For instance, after a CWP update, all hosts could no longer navigate to their associated “/roundcube” domains to access webmail. The problem was reported on the forums; however, it was left unanswered.
Untemplatable. I don’t only use my VPS to run my web services and applications, I also allow other people to host their websites, services, and applications on my VPS. When they login to their web panel, however, I want them to be reminded that they are using my VPS. Thus, branding the web panel is essential. CWP supported this in no friendly manner. Most web panels have a sort-of template feature, where one is able to define their own template for the web panel to use. However, CWP has no such thing, and even when looking for source images to replace and source files to edit, nothing can be easily found. If something is found, it is obfuscated. Most people would consider this to be a minor issue, however, the default CWP template is slow and unresponsive. The unresponsiveness was probably the straw that broke the camel’s back, so to speak.

With all of these small issues combined, and with a growing favoritism toward Linode’s VPS services, I decided that it was once again, time to make the transition to a new stack of services. This time, I decided to use Linode’s VPS plan, run CentOS7 with the LAMP stack, and finish it off with the free and open source Sentora web panel.

The Process

Preparing the VPS

Setting up the actual VPS with Linode is quite simple. I went with the Linode 4096 plan (Which is the same price as a Digital Ocean plan with less features) and deployed a CentOS7 image onto it. From there, after the machine is booted, the “Remote Access” tab provides all the necessary information to SSH into the machine and install the necessary software.

Installing Sentora

The Sentora installation process is fairly straightforward and fairly well documented. Setup with a fresh new Linode VPS, all of the requirements regarding a minimally installed machine were already met. However, installing Sentora requires a valid subdomain to be pointing towards the server machine. This subdomain will later be used to access the Sentora web panel.

At the time of installing Sentora, I did not have an extra domain lying around to point to the new machine, and knowing very little about DNS and DNS Records, I eventually discovered that what I needed to do was create a new “A” Record that pointed a subdomain of an already existing domain to the new machine. I edited my DNS Zone file to contain a record that looked as such:

www.panel. 1800 IN A 123.456.789.246

Which mapped the “panel” subdomain to the IP Address (123.456.789.246) of the new machine. After waiting about 15 minutes for this new DNS information to propagate, I was able to ping panel.domain.com and receive responses from the new machine’s IP Address.

Note that at the time, this process was two-fold: Since I was transferring from DigitalOcean, I changed my DNS Zone File on my DigitalOcean host to contain a NS entry that told the DigitalOcean host to “Look for the server that is serving the panel subdomain here”. That entry looked as such:

panel 1800 IN NS 123.456.789.246

Then, now that the DigitalOcean host knew where to look, the “A” record was added to the Linode host, which basically said “The IP Address for the panel subdomain is this”.

And with that, the pre-installation steps of Sentora are complete. The real installation can begin. As per the documentation, this includes downloading and running a Sentora installation script. The installation script is extremely straightforward and asks only about timezone, hostname (the subdomain that was just declared), and IP Address.

Configuring Sentora

I decided that the best user-architecture for my server would be one that treats every domain as an individual user, even the domains that belonged to myself. Although I could have chosen to attach my domains to the admin account, I wanted to create a separation layer that would allow each website to have its own configuration. On top of this, configuring my websites as “clients” of my website hosting would allow me to better support those who are actually clients of my hosting.

Thus, I added two new reseller packages that would take care of this. The BasicUser package was limited to 10 domains, subdomains, MySQL databases, email accounts, etc, but was given unlimited storage and bandwidth. I figured that 10 was more than enough for the clients of my web hosting. I also created the ElevatedUser pacakge, which had unlimited everything. This, of course, would be for my own domains.

Before moving on to actually creating the users, however, I wanted to customize the Sentora panel so that my brand would be reflected across the panel for my clients. I am a fan of the base Sentora theme, so I decided to basically use that theme, but replace all Sentora images with my own logos. This was done in the following steps:

Copy the “Sentora_Default” folder (The base template) to another folder (Your new template) in /etc/sentora/panel/etc/styles
Replace all images in the img/ folder with your logo
Replace information in /etc/sentora/panel/etc/static/errorpages with your information
Continue searching for things to replace in /etc/sentora/panel/etc

Then, once everything is replaced, the new theme is visible in the Sentora theme manager. Select it and voila!

User Creation and Migration

I created each user’s account using the Sentora admin panel. I would setup a generic username and password for them to use, and before alerting them of their new account information, I would do the following:

Login to the account and create a new domain, which was their domain.
Edit the DNS Records for that domain (The default DNS Records are perfect)
Add a new FTP account (with a generic username and password), set to the root directory for their account.

These three things would allow the user’s information to be transferred from the old server to the new server relatively easily. Transferring users is also a relatively easy process. First, I logged in as the root SFTP account on my old DigitalOcean/CWP configuration and downloaded everyones’ files onto my local machine. This, of course, took some time and allowed me to discover that some of my users were hosting a lot more than I previously thought they were (one user was hosting a 10GB MongoDB Instance). I also was able to go through and delete some of the files that I didn’t think would be used anymore, such as log files, backups from my previous server migration (From HostGator to DigitalOcean), and old files that were not in use anymore.

Once all of the files were downloaded to my local machine, I went ahead and uploaded all of them using the newly created user FTP accounts, being sure to put all of the public_html files into the new public_html directory.

I also logged into phpMyAdmin using the root login credentials. This allowed me to find all MySQL accounts attached to each user. For each user, I logged into the Sentora web panel and created MySQL databases with names matching the old database names. Although this could have been accomplished by simply exporting the tables, exporting the tables would not allow users to see their MySQL databases in their Sentora panel. Thus, creating the database in Sentora would allow the users to see their MySQL databases in the panel. I also created MySQL users for each existing user on the old MySQL servers. Unfortunately, Sentora doesn’t allow users to me named in the same format as my old setup, so I had to change the naming format a bit.

Now that all the databases were created, I could go into the old phpMyAdmin and export all tables using phpMyAdmin’s EXPORT functionality, taking care to make sure that the “CREATE TABLE” statements were included. I saved the .sql files, and then used phpMyAdmin on the new Linode host to import these tables.

WIth all database information transferred succesfully, the final change I made was establishing users’ subdomains. I was able to look through the DNS Zone file on the old DigitalOcean/CWP host and find the subdomains for each user. I then logged into each user’s Sentora panel and created the subdomain using Sentora’s Subdomain functionality. Unfortunately, Sentora’s subdomain functionality only creates a directory for that subdomain, so I also had to edit the DNS Information, adding a CNAME entry for each subdomain that was to be added. The CNAME entry looked like:

subdomain IN CNAME 123.456.789.246

More Database Migration

The most glaring roadbloack with transferring MySQL databases is that there is no way to actually view the password of a user. Although this is a great security feature, it causes problems when migrating servers becasue one is unable to view the passwords of old users to put them into new users.

The solution to this problem that I chose to use utilized mysqldump. When run from the old VPS host, this tool would dump all the important data pertaining to the old MySQL databases, including users and passwords. The full command looked as such:

mysqldump -u root -p mysql > mysql.sql

After typing in the MySQL root user’s password, this creates a mysql.sql file that can be downloaded and then uploaded to the new host, where it can be restored using

mysql -u root -p mysql < mysql.sql
UPDATE user SET password=PASSWORD('<rootPassword>') WHERE user='root';
FLUSH PRIVILEGES;

This command restores all of the users, their passwords, and their permissions to the server. The only problem with using this command is that, for me at least, it changed some information of important users such as the MySQL Root user, Postfix user, ProFTPd user, and Roundcube user. This was due to the fact that those database users just happened to have the same names as they did on my previous server. So, of course, they were overwritten. Thus, with that being said, the above command also contains a line to keep the root MySQL user password in check.

Luckily, Sentora stores a text file that keeps all of the default MySQL user passwords in /root/passwords.txt. Thus, using this file as reference, I went ahead and updated all of the users passwords accordingly. For example:

mysql -u root -p
UPDATE user SET password=PASSWORD('<MySQL Postfix Password>') WHERE user='postfix';
UPDATE user SET password=PASSWORD('<MySQL ProFTPd Password>') WHERE user='proftpd';
UPDATE user SET password=PASSWORD('<MySQL Roundcube Password>') WHERE user='roundcube';
FLUSH PRIVILEGES;

After these updates, everything on the system was working properly. However, I will admit that it took me a long time to figure out that importing my mysqldump file actually changed these passwords.

And with that, all MySQL user accounts were restored and working!

Small Adjustments

Now that all the major elements were transferred from the old server to the new server, I sat back and casually navigated to as many parts of the websites as I could. Mostly everything worked. I also followed certain steps to transfer MyBB instances. Those steps can be found here.

Another issue I was having involved a PHP script I had written to get the name of the song that I was currently listening to (via Last.FM) for my blog. For whatever reason, the script was not executing and was simply acting as plain text. After several hours of troubleshooting, I realized that the default Sentora installation disallows PHP’s short tags (So <?php is required instead of <?). This issue was fixed by editing the php.ini file in /etc/php.ini and seting short_open_tag field to “On”.

Finally, since I thought that some of my users would dislike navigating to panel.mydomain.com when they wanted to edit information about theirdomain.com, I created a simple HTML file with a Javascript redirect, called it index.html, and put it in the public_html/panel directory for each of the primary domains being hosted on my server. In this way, theirdomain.com/panel would take them to the right place and not force them to type my domain into the web browser.

The Final Step for users’ websites was for them to direct their domains to the new host. I directed users to use their domain registrar to register nameservers for their domains, ns1.domain.com and ns2.domain.com and have them point to the new host IP Address. Then, they directed their domains to the newly registered nameservers of the same domains. With this, the domains now pointed to the new host and all DNS configuration changes made on the new host would be picked up and reflected in the domain.

The Final Moment

With that, the migration was complete. In order to ensure that everything worked properly, I shut off my DigitalOcean host and went to bed. Thankfully, when I woke up in the morning, all websites still worked and pinging them returned a fancy IPv6 IP Address.

I ended up encountering some issues with SSL, Antivirus, and Malware later on, but I will cover those in a later post.