socrates

socrates is a command-line Python program I wrote to assist graders posed with a large number of student submissions to grade. It was also created out of the realization that graders often waste time with repetetive tasks, especially when using a GUI environment, to open student work, inspect it, test it, and eventually construct a grade file.

I began development during the summer of 2014, while Dave Sullivan was drawing up plans to adapt Harvey Mudd College’s successful CS 5 course at BU for the coming fall semester of CS 111. Previous versions of CS 111 were taught using Java, but this new course was based on Python. I knew that Python, being a high-level language, provided an inspect module, which allowed a running program to import Python code and access various properties of its defined variables, functions, and classes. This provided the initial inspiration to develop a Python program that could import student code and run a set of predefined tests.

Early in socrates’ development, I knew I wanted to allow a grading coordinator to be able to specify tests that graders would then run, with the following goals:

  1. Tests should be easy to write: grading coordinators should not have to write any Python code when specifying what should be tested.
  2. socrates should be complete: it should be able to automate as many tests as possible.
  3. socrates should be transparent: a course’s instructor should not have to write assignments in a way that makes them easily testable.
  4. socrates should be extensible: support for new test types and recognition of new file types must be easily implementable.
  5. Use of a grader’s own judgement should be minimized, so as to limit the amount of unfairness in grading.

The extensibility goal came out of the nature of Harvey Mudd’s CS 5 course, which, as a “survey” of computer science topics, included the use of different software in addition to Python, including Logisim for teaching circuit design and Picobot, a game-like, rule-based model of computation inspired by cellular automata.

socrates currently satisfies most of these goals, and was mostly feature-complete by fall of 2014. I’ve continued development on socrates, adding support for more file and test types, support for automatic determination of late penalties, a “batch” mode, and other improvements.

socrates is open source code, which can be found on GitHub. I’m currently maintaining its GitHub wiki, where I’m adding guides on how to install and configure socrates, as well as write criteria files.