Reclaiming Jekyll

Reclaiming Jekyll thumbnail

In December, I moved off of WordPress to Jekyll. This is easy to do with GitHub Pages, but I wanted to self-host because keeping a SSL certificate was important to me. I followed Tim Owen’s sample had everything up and working well.

I faced two specific challenges, though.

  1. FTP and SSH uploads were blocked in several places where I normally work. That meant I needed to remember which files needed to be uploaded via cPanel every time I wanted to publish a post. I would often forget an image or have a broken link, which meant regenerating the entire site.
  2. Because SSH was blocked, I had to use a cron job to publish the post. I would set one up to run every 5 minutes while I was working to make sure the changes were correct. Then, I would delete the cron job.

The bigger issue was that building on the server duplicated the site files. So, I’d have a folder of all of my posts and assets (images, styles, etc) that would get copied into the live site. Instead of shrinking my server footprint, it was doubled. No good.

My next idea was to use git, which is preinstalled on Reclaim shared hosting (which is awesome), to manage all of my files. But, I ran into the SSH problem again (HTTPS doesn’t work out of the box with git and setting it up is a headache). I also had problems tying the folder to the Reclaim location for some reason. So, that idea was out.

I continued to think about the problem and I finally landed on it: I wanted to keep all of my files on Reclaim when I really only needed the _site directory. I can build it on my computer and then publish only the live components.

This introduced another problem: it’s more complicated than just uploading the new post. The _site directory is changed and paginated with each build, so all of the relative links have the potential to change. How would I limit my upload to the site directory without needed to build on the server?

It turns out that you can pull single directories from a GitHub repo online. The key is only checking out the directory you want. Instead of using git pull to fetch and merge everything, you break it down into several steps.

  1. Set up an empty repository using git init.
  2. Assign a remote repo via url using git remote add . So, mine is called nodes-site and maps to https://github.com/bennettscience/nodes.git.
  3. Fetch the entire project with git fetch nodes-site. This finds and maps the entire project to git but doesn’t actually add any files yet.
  4. Check out a single folder with git checkout nodes-site/master -- _site. This creates a read-only directory!

I don’t need to write any files on the server…I do all of that on the computer. This step just grabs what’s been published to the Github repo and displays it as a live page on blog.ohheybrian.com.

Here’s my new process:

  1. Write and build the site on the computer. It runs faster, no need for the Internet.
  2. Use git to track all of the changes and push it up to GitHub. All of the files are public already through the blog, so I don’t care that it’s available publicly there, too. In fact, it serves as a nice backup in case I really bork something up.
  3. Write the steps above as a cron job to pull the _site directory in a couple times per day. If nothing changes, no new files are copied over. If there’s been a new post, that’s reflected in Git and the entire directory restructures with the changes.

My original folder (with everything) came in around 300MB. The new folder, with only the published material, is about 180MB. So, I saved 50% of my disk space by publishing only the live pages.


This StackOverflow post got me moving in the right direction.

Featured image: Allen-Bradley Clock Tower flickr photo by kallao shared under a Creative Commons (BY-NC-ND) license

Moving from WordPress to Jekyll

This is a long post in several parts. Jump to different sections using the links below.


Introduction

Long story short, I moved from self-hosted WordPress to a static HTML site generated by Jekyll.

WordPress does it’s job really well. I think there was some statistic [citation needed] that showed nearly 30% of the Internet runs on WordPress in one form or another. That’s a lot of the Internet.

But, because of ubiquity, there is a significant drawback: WordPress sites are prime targets for malicious code and hacking. A plugin on my site shows how many dozens (and sometimes hundreds!) of login attempts there have been. It’s a battle to make sure security plugins are always up to date. That leads to other issues: incompatibility with plugins.

So, This entire blog - 2018 all the way back to 2010 - is a set of static HTML pages generated by Jekyll on my Reclaim Hosting server. No more logins, no more plugins to check and update. Just nice, clean, lightweight HTML.

It took me several weeks to work out the details for the migration. It wasn’t too bad, but I learned some things along the way that I’d like to share here.

Exporting WordPress

Jekyll uses Markdown and YAML data to generate a website. It’s quite clever how they pulled it all together, actually, to mimic a typical dynamic (database-driven) blog like WordPress. There is a plugin which will export your WordPress blog formatted for Jekyll, including all post metadata like tags, permalinks, and image resources. It gives you a .zip file which you can then extract and use to populate your new Jekyll site.

First, it extracts your entire media library. WordPress automatically generates several files for each image you’ve uploaded for different display situations. My media folder was well over 300 MB because I didn’t take the time to clean the library up. I’d suggest cleaning up any unused image files before the export.

Second, any pages you have on your site (not blog posts) get their own folder. Take time to go through each folder and make sure it’ll fit the Jekyll file structure.

Finally, do a regular WordPress XML export so you have an entire backup of your blog. The Jekyll plugin only converts posts and pages. If you have other resources, you’ll want to save them somewhere before deleting or redirecting your site.

Hosting

The great thing about Jekyll is that it is ready to go with GitHub Pages. If you’re already using GitHub, you can go that route with your username.github.io account with a single commit and push. I have a lot of traffic (humblebrag much?) to blog.ohheybrian.com already and I don’t want to set up a redirect. I’m also already using my GitHub Pages site for my (web apps)[https://dev.ohheybrian.com]. You can map a custom domain to GitHub Pages, but you cannot use HTTPS on that domain, which was a dealbreaker for me.

Each web host is different, so you need to see if yours supports Ruby 2.4 or higher. Lucky for me, Tim Ownes from Reclaim Hosting already had a blog post on setting it up with my Reclaim account. I followed his instructions to the letter and got it working on the second try (I borked some theme and file structures on the first, so I deleted everyting and started over).

SSL is a big deal. If you don’t know what it is, read The Six Step “Happy Path” to SSL by Troy Hunt (or anything else he writes, honestly).

Comments

I don’t get a ton of comments, but with a static HTML site, there isn’t an easy way to include comments. If you’re hosting with Github Pages, Staticman is an awesome and secure way to include comments on your posts. Another option would be to use a third-party tool like Disqus. I didn’t go with Disqus because they’ve had some trouble with clean use of user data in the past.

I decided to create a custom commenting form (based on this post) using Firebase. It’s a real-time database run by Google which can push information anywhere I want it to go. Each post has a query to the database to check for comments. Pushing the comments to the database is handled with a little bit of JavaScript, which I’ve modified from the linked tutorial:

Firebase also includes cloud functions that can be written in Node JS. I’ve never written any applications in Node, so this was a learning experience for me. This function watches the comment database and simply notifies me if a change has been made using the following script:

It could definitely use some refinement, but it does what I need it to do.

Updating

Relying on an Internet connection to write a blog post seems so 2012. With Jekyll, I can write in any text editor and then upload when it’s ready. If I’m on my main machine, I can even serve the page locally to see what the update will look like as if it were live on the web. It’s a small perk, but as I’ve moved to working more and more with text files (rather than things like Google docs) it’s nice to be able to open a blank text file and start writing. I can come back whenever I want and finish up or let it sit on the large pile of started-and-never-finished posts.

Conclusion

In the end, this is a highly technical shift away from something built for the end user into something I have absolute control over. If the blog breaks, it’s my fault and I will have to work to fix it, which is satisfying in its own nerdy way. It’s definitely not the easiest route to start (or continue) blogging, but it’s mine, which is fulfilling.

If you’d like to know more about how to make a switch, feel free to try out that nifty commenting section below or just write me an email: brian [at] ohheybrian [dot] com.

Featured image is Find your way flickr photo by maximilianschiffer shared under a Creative Commons (BY-ND) license

Building a Custom Registration Form with Google Apps

One of the main responsibilities my team has is offering and running training on a number of resources teachers in the district have access to. The most popular, because of their new-ness, are Google Apps (GSuite…whatever) and Canvas. Because we offer so many workshops, our Drive is full of copies of copies of copies of registration forms. So, I built a custom registration form using Google Apps Script and a Sheet. This detail-heavy post walks through the entire process with code.

The End Product

There were a lot of iterations of the project and in the end, we wound up with a website which:

– Can be managed by anyone on the team

– Collects the username (their email address, actually)

– Displays available seats for the workshop

– Allows users to both register and cancel registrations

– Displays the user’s current registrations

The Back End

This uses a Google Apps Script web app and a Spreadsheet to run the back end service. The sheet serves as two databases: course offerings (with details) and user registrations. The front end site uses AJAX calls to populate a page specific to the user.

In order to return the correct data, Session.getEffectiveUser().getEmail() is used to check the logged-in user against the database so only their course information is returned. For example, this script gets the current user registrations from the registration sheet:

Much of the backend work is building arrays and objects of the correct data and sending it to the client side scripts to display. The full commented code is in this GitHub Gist.

The Front End

Google Apps Script allows for templated HTML. I’m not using a template, per se, because it’s loading a static page and populating div elements with AJAX calls. But, the same could be done to create multiple pages using the same template.

There are two main actions: registering for a course and cancelling a course. Both call the server to make changes to the spreadsheet and pass updated information to the front. Registering for a course (or courses) write across a range rather than submitting n number of results for that user. Cancelling a course, instead of building and writing a new array, searches through the Sheet for the user and the matching date and simply deletes the cell. The app is then reloaded to refresh the user registrations at the top.

It’s not pretty and there are some changes I still want to make to clean up some of the array building/passing, but it’s such a small amount of data, the site loads very quickly even with a large number of registrations.

If you have suggestions, or if you make improvements, please leave a note in the comments. As is, you could probably copy and paste most of this into your own sheet and get it going. Be sure to fill in the correct sheet names and ID’s in the code.gs file to avoid breakage.

Networks flickr photo by eflon shared under a Creative Commons (BY) license

Easily Filter Large Data Sets in Google Sheets

I use Google Sheets in my classroom a lot. I used to rely on combinations of the vlookup, importrange, and index/match functions to get information, but I’ve recently switched to using if and filter to return arrays of information from master spreadsheets.

Using filter is nice because it takes multiple conditions and you can set which columns of the array you want to return for your summary sheet. A pretty standard search looks like this:

=if(filter($A$2:$A,$A$2:$A=$F$3)=$F$3,filter($B$2:$B,A2:A=$F$3),"")

Here’s a sample spreadsheet so you can see how the result is returned after changing the filter term.

So, let’s break it down:

Cell F3 holds my search term, “A”, “B”, or “C”.

(filter($A$2:$A,$A$2:$A=$F$3) – Filter looks through a range of cells ($A$2:$A) for a specific condition ($A$2:$A=F3), much like the IF statement. The exception is that this only returns the matching content rather than a boolean (true/false). The filter, in this case, is serving as the boolean check for the IF statement it’s wrapped within.

=If() – This function is super helpful because it limits what happens in the sheet. It’s like conditional formatting, but for your functions. It takes two arguments, minimum, but you can set up to three: the condition to check, what to do if true, and what to do if false. In this case, the conditional is set with the filter function (see above). If the filter returns a cell with an “A” in it, the TRUE condition is run.

filter($B$2:$B,A2:A=$F$3) – If it’s true, I want a different column returned. In this case, it’s the names of students with group “A” set. Filter works the same way, except this time, it searches for column B (the names) that match the search parameter (“A” in column A).

In other words, the function reads like this:

  1. If

– filter through column A

– Look for cells that contain “A”

– If an “A” is found, the IF statement is TRUE

  1. Execute the “TRUE” parameter

– Print the student’s name in the cell

  1. If not, leave a blank cell

I added a third column, which prints the student’s project content just to show how these functions can be used in conjunction with one another.

I know you can use ARRAYFORMULA to do essentially the same task, but using ARRAYFORMULA doesn’t allow you to add custom content in the column – the throws an error saying data cannot be overwritten. I don’t run into that case often, but it’s often enough to be annoying.

Again, this is difficult to see without checking out the example spreadsheet. It’ll take some playing, but once you get it, it’s very helpful. Leave a note in the comments if you get stuck and need some help working it out.

How To Get Text Messages into Google Form Responses

Continuing on my data collection and analysis streak, I’ve got a little project to throw out into the wild and see what happens.

Student response systems are expensive and can be clunky. Google Forms are taking over the role of the stand-alone response system because of their flexibility in the wild during collection (device agnostic) and during analysis.

I like using forms, but they don’t work so well when students don’t have access to an Internet-ready device. So, I came up with a way to allow students (or anyone, really) to respond to a poll using either a form input or a text message.


The challenging part of this is getting the message into the spreadsheet. I have a Google Voice number, so I looked into Apps Scripts for Voice. Unfortunately, there is no way to access Voice using Google Scripts. There is an API for Hangouts, but it’s not documented very well and doesn’t look like it would even work with Apps Scripts or Spreadsheets. Maybe I’ll head back down that road someday, but not right now.

I decided to go with IFTTT, which has some benefits (easy to set up and manage) and drawbacks (only checks every 15 minutes-ish, so you need to manually run it during a session). It’s clunky and adds a breakpoint in the flow, but it works.

The Setup

You need to start by making a Google Form. For this, I want to allow students to send in four things: Their name, a multiple choice answer, a confidence rating, and any comments. Create the form with your collection in mind.

In IFTTT, you have to tell the script which cells to populate in the sheet with three straight bars, “     ”. Knowing your form setup is really important before you make the IFTTT recipe because it’ll save you a headache later. My recipe puts the responses into the first three columns and then I use a Google Apps Script to break them into the correct columns.

The Flow

Method 1 – Pure Google Forms

The audience opens the Form and answers the question displayed live. They hit submit.

Easy.

Method 2 – Text Message to IFTTT

First, in Voice, you set up email notifications for text messages. Then, in Gmail, set up a filter to catch the emails and do two things: add a label (I used “text”) and then mark it as “read” (mostly for your inbox-zero sanity). Now, have the audience send a text to your Google Voice number (see below for formatting).

In IFTTT, set up a trigger which uses Gmail -> Google Drive. The Gmail trigger should do a search for your new Gmail label. When it finds that email, it appends a row to your form input spreadsheet.

2015-04-30_21-38-48

More complicated, but now we have a way to get raw text data into the spreadsheet. Now, you need to make sure the text message is formatted correctly.

Splitting the text

Google forms split fields into their own columns. You can’t do that with a text because IFTTT puts the whole message into one cell. To split it, you need a way for a script to find the individual components. For this, I have participants do the following:

Name [comma] MC answer [comma] Confidence (1-4) [comma] comments

IFTTT dumps this into a placeholder cell in the spreadsheet and it can now be broken up at each comma using the following script:

splitsms

function splitCells(from) {
  var sheet = SpreadsheetApp.getActiveSpreadsheet().getSheets()[0];
  var colC = sheet.getRange(2, 3, sheet.getLastRow()).getValues();  // C2:C, only non-blank rows
  //Logger.log(colC);
  for(var i=0; i< colC.length; i++){
    if(colC[i][0] != ''){
      // If we received a SMS response, set a formula to parse it
      sheet.getRange(2+i,4).setValue('=if(istext(C2:C),split(C2:C,",",true),"")')
    }
  }
}

After running this script, you now have all the responses in the correct column and you can play with the data more and look for patterns in the responses.

    • *So, there’s the start of the idea. I have a second sheet going, but I’ll write that up in a subsequent post. If you’d like to try this yourself and you’re having trouble reading my rambles, here’s the tl;dr version:

    1. Get a Google Voice number.
    2. Turn on email notices when you receive a text message.
    3. In Gmail, set a filter for your email notifications.
    4. Sign into IFTTT and copy this recipe.
    5. In the spreadsheet, copy and paste the Google Apps Script above, which splits the text message rows into columns, into the script editor. You may need to refresh the sheet to have the menu added to run the script.

I’ll follow this up with the data analysis sheet explanation.

Making Endnotes from Footnotes

**Update 12.22.2016** Since this has become quite a popular script, I’ve converted it into a Google Docs Add On so it can be used across documents. Check out [the blog post](http://blog.ohheybrian.com/2016/12/introducing-endnote-generator-add-on/) or the [Endnote Generator website](http://dev.ohheybrian.com/endnotes) for more information.


**Update 8.27.2016** As Phil noted in the comments, the script didn’t format anything over two digits in length. That has been [corrected in the source](https://github.com/bennettscience/footToEnd). If you need more (100+ endnotes may be too many, though…) leave a comment and I’ll go back and make the changes.


Krissy Venosdale asked on Tuesday if there was a way to make endnotes in Google Docs as easily as footnotes. I thought, “Surely there is.”

Well, there’s not.

So I wrote a script that does it.

It’s hosted on GitHub along with instructions on how to install the script (it’s not an add-on…just a custom script) and run it in your document.

Here’s a peek:

When you add the script to the document (manual process right now – read the instructions) it adds a custom menu from which you run the script.

I know endnotes and footnotes aren’t really used in high school, but if you’re writing a book or something…maybe it’ll be helpful to you.

Either way, it was fun to write and it works pretty quickly. I haven’t tested it with documents more than a few pages long, so if you’re going to try it on something major, I’d suggest making a duplicate copy to run it on first, just in case.

I’m a teacher by day, after all.

Making Sub Plans Easier

I hate missing school. It’s stressful. I have to do multiple days worth of planning ahead of time, plus know what’s happening the day I get back. Then, I have to turn those plans into a written document that anyone in the world can follow to the letter.

It’s stressful.

I still leave a document with some specific notes for a substitute – things like management or specifics to take note of. But now, instead of writing out a ton of information on what to do, I record a simple video and post it to a static page on my class website. There are a couple little tricks I learned along the way:

1. The page should be bare bones. Don’t worry about making it “feel” like the rest of your class site. It should be something easy to display on a projector without having to hunt through menus to get the information. That also allows you to just replace the video and written instructions for the next time you’re out. Flexibility is key.

2. Embed the video right in that page. If you’re like me, you want to make sure that video is responsive, too. I found a super-handy little snippet of CSS to make YouTube videos responsive, and I used that so students can access the information on their own later on any platform.

3. Keep your video unlisted so it stays within context. I have a lot of content on YouTube, I want to keep the sub plans isolated because they don’t really fit in the flow of anything else there. This way, the video only “exists” when student need it – while I’m gone.

It’s not totally un-stressful now, but it’s way better than it used to be. If you’d like to see my “away from school page,” or even copy the source for your own page (self-hosted), that’d be dandy.

How To: Find Emails You Haven’t Replied to in Gmail

David Wees asked a question on Twitter today that I thought would be pretty easy to solve:

Turns out, no so much.

In my searching, I did come across a service which reminds you to follow up with emails which are unreplied to you, but not ones which you need to send the reply.

You can use filters and other little workarounds to get it done, but that usually means having to remember to actually add the filter or mark it unread when you’re going through your inbox. That’s hard to remember to do in the moment, and can even be impossible (like if you’re using the mobile app). Luckily, you can combine search operators in the web view to get pretty good results.

The trick is you have to be using a signature so that every email you send has a unique string in it that we can filter out.

For me, I have a link to my website – ohheybrian.com. It’s pretty unique, which makes it a good search term. You can use your name, too, but you want to be careful with that in case the sender used your full name – if that’s the case, it’ll get filtered out.

To find emails which need replies, you want to use is:read -Re: -your_unique_phrase.

This checks for read emails (in case you forgot to mark it unread), without Re:, and without your_unique_phrase.

Mine reads: is:read -re -ohheybrian.

Emails which are part of a thread still show up, but they’re based on the last email of the conversation. So, the thread may contain replies, but you haven’t replied to the last one you received, which can still be handy.

You can limit it even further by using in:inbox or some other filter, but that isn’t usually necessary.

Unfortunately, this isn’t perfect, so you’ll still need to do some spot checking. There is no perfect solution as of right now (no is:reply filter or something similar) which you can use, but Google does have a good list of the Gmail search operators that you can play with to build some pretty powerful searches.

More Attribution Done Just for You

I’m expanding on a post I wrote a week or two ago in which I added automatic Flickr attribution to header images on the blog theme I’m working on. I wanted it to be done on all images on the blog, and I finally got the script after playing around and with some help from StackOverflow. Here’s the skinny:

I didn’t expand my original script – I want that one to run on its own because it styles the credit a little bit different than the body text. Rather than overlaying a credit, which would require some HTML restructure, I’m simply adding it below the picture because KISS is always the best policy.

Here’s a CodePen demo of the script in action.

A couple things to note about the script:

  1. Right now, it only adds a credited caption to Flickr photos, because, let’s be honest: they have the best API for this kind of thing. Don’t hope for anything like this on Instagram any time soon.
  2. It specifically looks for the Flickr URL before running the script, so your site won’t be bogged down with scripts running.

So, there it is. Take it as it is, or take it, mess with it, and share it back.

Also note I’ve got a larger project going which will get its very own post someday soon coming up.

If You’re Not Going to Use Attribution, I’ll Just Do It For You

It’s important to use attribution online. When you pull a picture, make sure you tell other people where it’s from. Some sites make it easy for you to do that, others…not so much. Flickr used to be great at making attribution easy, but their latest redesigns have made it harder to accomplish. Alan Levine has a great CC attribution gizmo which makes it super easy to both give credit to and share a photo from Flickr. I use it all. the. time.

WordPress is great – I use it for this blog and my class websites. I even ran my own multi-site for my students one year using WordPress because my administrators at the time didn’t want blogs out in the wild. So, I did it for them. Anyways, WordPress doesn’t have the best method for featured images. You can’t use an image source URL to share it…you need to download the image and then reupload it to your site. It’s really easy to “forget” to post attribution.

I’ve been playing with another open source CMS called Anchor. It’s very bare bones right now, but it’s super flexible in terms of what you can do. Because I didn’t have enough going on (read sarcasm there…) I decided I wanted to create my own custom theme for an Anchor blog. I’ve got a demo site up right now (which will be changing soon…updates are coming!) and one thing I had to include was featured images from a URL.

The next step was to get attribution in there…not just a link back, but an actual box of text with the post title at the very least. Flickr has a powerful set of APIs which can be used to get all sorts of data which you can then bend to your will.

For those too wary for code, here’s the tl;dr – this new blog theme in Anchor will automagically attribute featured images you include. All Flickr images are coming soon.

The problem – In order to use the Flickr API, you need – at the very least – a photo ID. Because I’m attaching the photo using CSS, I have the source URL of the image. Here’s an example:

I needed a way to get that snippet of the URL into the API call. Unfortunately, there is no one step that can do this. So, I went and gave myself a crash course in regular expression to get that information into a usable form.

Of course, there’s another issue in my way. The CSS holding the URL for the image has characters in the syntax which I can’t use:

The solution: use another regular expression snippet to pull it out.

If you’re following along, if you put console.log(bg) into your editor, it’ll return a clean URL.

Now that we have the URL, it’s time to extract the photo ID only because that can be used in the Flickr API to build the attribution URL.

Like I said earlier, I gave myself a very crash course in regular expression, so this very…very ugly expression1 strips everything except for the photo ID and stores it in a variable for later.

OK, we got the URL, then we got the photo ID. Now, it’s time to build the URL to request information from Flickr about this picture.

This is the URL that we use in the final step of the process to get information from Flickr and then build one more URL, which becomes the attribution link.

This last asks for JSON information from Flickr and then we use jQuery to apply it to a div created in the HTML to hold the information. Flickr URLs all have the same structure, so building a link back to the owner’s page is easy. I just pulled out their user ID number and reattached the photo ID we grabbed earlier.

If you want to play with it yourself – changing the photo and everything – you can do that in this CodePen demo I set up during testing.

This is a lot of work to automate the link backwards, but hopefully, it’ll make it easier to add attribution back with every picture, not just ones you remember to grab information for. Again, this only work with Flickr at the moment, and only for featured images at that. I’m planning on expanding this to any image in a post pulled from Flickr as soon as I have time. Or, you could do it. Just share it back when you do.

1. no one said code had to be pretty to work