Steven Hensey

Educational Technology

Problem Description:
Rethinking How Schools Work: Creating Personalized Student Learning
In Elementary Education (K-5), how can we take a well-defined set of learning standards (Common Core Math & English Language) and efficiently track each individual student’s progress in each required skill to provide a more personalized learning experience? (Either reinforcing or challenging certain skills as needed.)

Tool Design:
Based on my problem, I propose creation of a tool that accomplishes two main goals:
1) Helps teachers efficiently track student skill progress in the Common Core curriculum
2) Provides a free, easy to use resource library to help develop each specific skill as needed

The end result should be a functional tool ready for teacher use. In order to create a manageable scope, the tool will focus only on the second grade so that I will have sufficient time to compile a relatively comprehensive library with regard to those skills. The library will ultimately be teacher-curated, but for this project I will populate the library based on teacher-suggested resources found through surveying. Teacher surveys will also be used to influence the design process by eliciting periodic feedback. If there is demand, the tool can be populated with the appropriate library resources to work with other grade levels in the future.

(*Work In Progress)

All Common Core standards have been loaded for grades K-5, in both Math and English-Language/Literacy.

Student skill tracking has been implemented to individually track progress towards each Common Core standard though an intuitive and responsive user interface.

What are Raven's Progressive Matrices?

Raven's Progressive Matrices (RPM) are a set of visual questions that measure reasoning ability / general intelligence. Each question is presented in matrix form, and my project addresses 2x2 and 3x3 problems.

The logic required to test a matrix of unknown 3x3 images, determine what relationships might be present, and find the answer becomes a balancing act between trying every possible combination against every test, versus obtaining a likely answer in a reasonable amount of time.

My 'Agent' is programmed for efficiency and is able to consistently solve 'basic' 2x2 and 3x3 problems correctly. Accuracy thresholds are set so 'likely' answers can be returned in cases where the Agent does not find a single correct answer.

Sample Problem: Basic E-07

Below is the verbose result for how the Agent correctly solves Basic Problem E-07. This is an interesting example because the 'IS_XOR' test finds a relationship between the rows, (A xor B == C) and (D xor E == F), yet it cannot determine an answer because of the overlap present of the circle in (G xor H).

When the Agent checks relationships by column, the overlap is no longer in the answer analogy, and the correct answer is determined.

---
Basic Problem E-07

>>> Checking by row for 'IS_EQUAL' <<<
A == B == C ?
Similarity: 0.409350824083 > 0.9 ? Density Error: 0.0671477878883 < 0.03 ?
Similarity: 0.483213053028 > 0.9 ? Density Error: 0.100721681832 < 0.03 ?
D == E == F ?
Similarity: 0.320973990039 > 0.9 ? Density Error: 0.042395336513 < 0.03 ?
Similarity: 0.514396749691 > 0.9 ? Density Error: 0.0388794567063 < 0.03 ?
Different

>>> Checking by row for 'IS_OR' <<<
A OR B == C ?
Similarity: 0.635054287614 > 0.85 ? Density Error: 0.364945712386 < 0.03 ?
False

D OR E == F ?
Similarity: 0.671823722457 > 0.85 ? Density Error: 0.309657759025 < 0.03 ?
False

>>> Checking by row for 'IS_OR_v2' <<<
A OR C == B ?
Similarity: 0.706182140483 > 0.85 ? Density Error: 0.293817859517 < 0.03 ?
False

D OR F == E ?
Similarity: 0.643269344011 > 0.85 ? Density Error: 0.333294076081 < 0.03 ?
False

>>> Checking by row for 'IS_XOR' <<<
A XOR B == C ?
Similarity: 1.0 > 0.8 ? Density Error: 0.0 < 0.1 ?
True

D XOR E == F ?
Similarity: 0.958207957205 > 0.8 ? Density Error: 0.0153794717486 < 0.1 ?
True

G XOR H == 1 ?
Similarity: 0.601273284603 > 0.8 ? Density Error: 0.286747393206 < 0.1 ?
False

G XOR H == 2 ?
Similarity: 0.386701249705 > 0.8 ? Density Error: 0.140453992704 < 0.1 ?
False

G XOR H == 3 ?
Similarity: 0.987974534308 > 0.8 ? Density Error: 0.217672016233 < 0.1 ?
False

G XOR H == 4 ?
Similarity: 0.976184862061 > 0.8 ? Density Error: 0.600094295144 < 0.1 ?
False

G XOR H == 5 ?
Similarity: 0.601273284603 > 0.8 ? Density Error: 0.26011863224 < 0.1 ?
False

G XOR H == 6 ?
Similarity: 0.386701249705 > 0.8 ? Density Error: 0.279966044143 < 0.1 ?
False

G XOR H == 7 ?
Similarity: 0.0 > 0.8 ? Density Error: 0.334640727957 < 0.1 ?
False

G XOR H == 8 ?
Similarity: 0.386701249705 > 0.8 ? Density Error: 0.335062485263 < 0.1 ?
False
Answers Found: [ 0. 0. 0. 0. 0. 0. 0. 0.]


>>> Checking by column for 'IS_XOR' <<<
A XOR D == G ?
Similarity: 0.986596583443 > 0.8 ? Density Error: 0.228820429672 < 0.1 ?
False

B XOR E == H ?
Similarity: 0.897833768628 > 0.8 ? Density Error: 0.0943309264096 < 0.1 ?
True

C XOR F == 1 ?
Similarity: 0.662975778547 > 0.8 ? Density Error: 0.0279179280188 < 0.1 ?
False

C XOR F == 2 ?
Similarity: 0.487889273356 > 0.8 ? Density Error: 0.146366782007 < 0.1 ?
False

C XOR F == 3 ?
Similarity: 0.937889273356 > 0.8 ? Density Error: 0.0621107266436 < 0.1 ?
True

C XOR F == 4 ?
Similarity: 0.965397923875 > 0.8 ? Density Error: 0.454974068835 < 0.1 ?
False

C XOR F == 5 ?
Similarity: 0.486332179931 > 0.8 ? Density Error: 0.0083044982699 < 0.1 ?
False

C XOR F == 6 ?
Similarity: 0.513667820069 > 0.8 ? Density Error: 0.018675721562 < 0.1 ?
False

C XOR F == 7 ?
Similarity: 0.231660899654 > 0.8 ? Density Error: 0.0931910887982 < 0.1 ?
False

C XOR F == 8 ?
Similarity: 0.487889273356 > 0.8 ? Density Error: 0.512110726644 < 0.1 ?
False

Answers Found: [ 0. 0. 1. 0. 0. 0. 0. 0.]
Final Answer: [0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0]
Basic Problem E-07 finished
---

Project Description

Given ten 60-second video clips (30 fps) of a robot moving in a wooden box, predict the position of the robot as accurately as possible for each of the following 60 frames (2 seconds) after the end of each video. Predictions are compared to the actual video data after the end of the video to calculate error.

Custom Filter Overview

I chose to keep my assumptions simple and see what level of error I could obtain.

Values to track during each update step:
1. xy-coordinates
2. distance traveled
3. distance rate of change
4. heading
5. heading rate of change

It takes 3 measurements to obtain enough information so that xy-coordinates, distance, and headings are all being passed along for each future iteration. My method was to use averaging of past distances and headings in order to obtain an accurate prediction for future movement.

In order for my filter to work, I had to first determine the extents of the world, or in this case the walls of the box that the hexbug was in, along with the center candle obstacle. I created a function world_size in helper.py that took the training_data.txt as input and calculated the xmin, xmax, ymin, and ymax for the walls. It also calculated the center and radius of the candle obstacle.

World Extent Calculations

Wall Calculations
The function simply creates 2 lists, ‘X’ for x values, and ‘Y’ for y values. It then iterates over all of the data points stores each x and y in their respective lists. At the end it returns min(X), max(X), min(Y), and max(Y) making the assumption that the world is a perfect rectangle.

Candle Obstacle Calculations
At first, the function assumes the candle is centered in the box. It checks for the nearest horizontal and vertical points from the assumed center by iterating through all values in training_data four times, one time for each value. This is accomplished by creating “pie-slices” from the assumed center of approximately 2 degrees, and checking for the point closest to the center in each of those slices:

The center of the candle may be offset:


2° slices used to find best values for horizontal and vertical variables:


After the best points have been found, the assumed center is shifted towards the actual center:
x_center = x_left + (xd₁ + xd₂) / 2 y_center = y_bottom + (yd₁ + yd₂) / 2

Bounce Calculations

After having calculated the world extents, I needed a function to check if the next estimated value provided by the filter collided with any of the walls or the center obstacle.

Walls
The code for colliding with the walls was fairly straight-forward.
1. Find the intersection of the line segment (from last_xy to new_xy) and the wall.
2. Determine the bounce_heading based on which wall had the collision
3. Take the distance between the collision_point and the original new_xy and go that distance along the bounce_heading to finally arrive at the bounce_xy.

Center Obstacle
The code for determining the center obstacle collision was a bit more involved, but with the help of WolframAlpha in solving for (x, y) of a line intersecting a circle I was able to create a function that bounces off the center obstacle.
1. The equation of a line and a circle has 2 intersections. Since only 1/30th of a second has passed, the new_xy point should be fairly close to the circle, so take the intersection point that is closest to new_xy.
2. Using a heading from the center of the circle to the intersection point, calculate the new_heading.

Wall Collision Example:


Center Obstacle Collision Example:

Functionality

Custom Filter Functionality:
Because the hexbug moves so erratically, using averaging over the entire data set would most likely degrade the results substantially, so I set a variable ‘f’ for the number of previous frames of data to consider. Through my testing it seemed that around 150 frames yielded the best results with my custom filter.

Topics Explored

(Please view the Papers for more detailed information on each set of experiments conducted.)

Supervised Learning

• Decision Trees
• Boosting
• KNN
• Neural Networks
• Support Vector Machines

Randomized Search

• Finding Neural Network Weights: standardizing data to increase algorithm efficiency
• Hidden Nodes: testing to find the optimal values
• Randomized Hill Climbing
• Simulated Annealing
• Genetic Algorithms
• Optimization Problems

Clustering and Dimensionality Reduction

• Expectation Maximization
• K-Means
• Dimensionality Reduction
• Clustering with Dimensionally Reduced Data
• Neural Networks on Dimensionally Reduced Data
• Neural Networks on Clustered Data

Reinforcement Learning

• Markov Decision Process Problems
• Policy Iteration
• Value Iteration
• Q-Learning

Machine Learning for Trading

Developed Learning Algorithms in Python for:

• Linear Regression
• KNN
• Bagging

Developed Strategy Learners for Trading Stocks:

• Started by coding a KNN and Q-Learner in Python from scratch.
• Coded a utility to read in historic stock data in .CSV form and parse / analyze various stock features using the Pandas library.
• Predicted 5-day future returns, with subroutines determining whether to enter, hold, or exit positions.
• Performed above SPY-500 both in and out of sample.

Project 1: Temporal Difference Learning

A replication of the results of Richard Sutton’s 1988 paper, “Learning to Predict by the Methods of Temporal Differences”. I will show that utilizing Temporal Difference (TD) learning methods can achieve more accurate results than standard supervised-learning (SL) methods. TD methods take into account changes in successive predictions, whereas SL only takes into account the change between the prediction and the final result.

The benchmark supervised learning method will be the Widrow-Hoff (WH) rule:


With WH, changes in weights are updated given a learning rate, but these changes can only be made after the final outcome (z) has been observed. The TD method used will be TD Lambda:


With TD Lambda, changes in weight no longer rely on the final outcome (z), but rather only on the next immediate prediction, and a weighted sum (by lambda) of all past predictions based on their eligibility. The direct comparison between TD and SL is made by the fact that the Widrow-Hoff rule is equivalent to TD(1), or TD(lambda)where =1. The problem utilized to show how TD(lambda) for lambda<1 consistently outperforms the SL Widrow-Hoff rule is a random-walk set up as follows:

• There are 7 total states: A, B, C, D, E, F, G
• D is the starting state, A & G are terminal states (reward at A = 0, reward at G = 1)
• During each step there is a 0.5 probability the agent will move right or left
• Once the agent reaches a terminal state, the walk is completed

Using Python, I created a random-walk generator that produced training sets as outlined by Sutton. Each training set consisted of 10 random-walk sequences, and I created a total of 100 training sets.

Project 2: Correlated-Q Learning

This paper focuses on replicating the results of the 2003 Greenwald / Hall paper, “Correlated-Q Learning”. Given four different multi-agent learning algorithms: Q-Learning, Friend-Q, Foe-Q, and Correlated-Q, I will explore the resulting policies learned when playing a zero-sum game that has no deterministic equilibrium policies. The game played will be a 2x4 variation of the grid-based soccer game described in Littman’s “Markov games as a framework for multi-agent reinforcement learning”.





Based on the “Correlated-Q” paper and the referenced Littman paper, I set up the game as follows:

• There is a 2x4 play-field consisting of player positions 0 - 7
• There are two players (A & B), both cannot occupy the same space at the same time
• The ball is always controlled by either A or B, and in the same position as that player
• The non-terminal states are when the ball is in any of the center 4 positions: [1], [2], [5], or [6]
• The terminal states are when the ball is in any of the goal states [0], [4], [3], or [7]
• The goals for A are [0] or [4], the goals for B are [3] or [7]
• If the ball is brought into a goal position (regardless of which player moves the ball), the owner of the goal receives +100 reward, and the opposing player receives -100 reward
• The actions available are N, S, E, W, and stick (no move)
• Both players choose an action and then a player is chosen at random to move first
• If the player currently taking an action (say playerA) would collide with the other player (playerB), then playerA remains stationary; however, if playerA had possession of the ball, then the ball is passed to playerB. (*You cannot steal the ball from a stationary player by colliding with them).
• When a goal is scored the game ends

A custom python solution using numpy and pulp was used for this project. The state space is set up as a set of triples: (positionA, positionB, ball_posession). Players A & B can occupy any combination of two positions, but not the same position, therefore the legal states are (0-7, 0-7, 0-1) where A_position != B_position. This results in 112 unique states. The tracked (state), (action) pair for plotting changes in Q values is (2,1,1), (1,4) → (A in position 2, B in position 1, B has the ball), (A moves S, B sticks).

The action space varies based on which learner is being used. The basic actions are 5 single values (0-4) where 0=N, 1=S, 2=E, 3=W, and 4=stick. For certain learners the action space is extended to a set of 25 doubles: (actionA, actionB), which represent all action combinations for both players.

Q-Learning Plot (non-convergence)

mVitals

The mVitals project is designed to improve patient care through better coordination and visualization of electronic patient medical records.

The overarching purpose of the project is to give the maximum patient overview in the minimum amount of space and time. By leveraging the existing XML-based Continuity of Care Document standard, mVitals supports the physician’s mental model by providing a friendlier, more visually impactful and ultimately more effective view on patient care information.

Using the already established CCD standards as its core recordset allows mVitals the advantage of maintaining broad interoperability with all providers adhering to the basic CCD model. The project’s coordination of these diverse recordsets gives all physicians involved with the patient a unified view of the patient’s medical history.

This comprehensive patient view will bolster any physicians ability to provide the best care for the patient, and moreover, minimize costly issues surrounding redundant care.