Standard

Posted by

Posted on

January 10, 2016

Posted under

Linux

Comments

1 Comment

Tcpdump: When and How?

A tool I rely on heavily for network debugging is tcpdump. This tool naturally comes to mind when I run into issues, but it may not for others. I thought I’d take a moment and describe when I reach for tcpdump and give a quick primer on how to use it.

If you’re using windows, windump/wireshark are the cli/gui equivalents. I’ll stick to tcpdump in this article, but many of the CLI options are the same and the filters are pretty similar if not the same.

When should you use tcpdump?

Whenever you’re troubleshooting an application, you hopefully have some sort of application-level logging to help you figure out what’s going on. Sometimes, you don’t have that – or what does exist provides inadequate detail or appears to be lying to you. You may also not have access to a device that you think is affecting the traffic, and you need to ensure that the traffic flow meets your expectations. As long as your application talks on the network, even locally, tcpdump may be able to help you!

You may have users from the internet who need to reach your application who are not able to, and they’re only receiving a timeout, but other users have no issues. You look in your web server logs and you don’t see any logs for the user complaining. There are log entries for the users who are not complaining. You can use tcpdump to listen on the webserver’s port for the customer’s IP and see if the connection attempts are seen. You can also see the packet contents in cleartext (as opposed to binary format – encrypted content is not decrypted, it’s just more easily visible) if that helps diagnose the issue.

Many applications also rely on local connections, typically on the loopback interface, and may be affected by the local firewall (iptables or the Windows Firewall Service, for example). Using tcpdump, you can see if the packets are immediately rejected, which is likely to be the firewall service, or if it completes a three-way handshake before closing the connection. In almost all cases, if a three-way handshakes is observed, the application has received the connection.

Given the name tcpdump, it’s worth nothing that you can see almost anything on the wire, not just TCP packets. UDP, GRE, even IPX are visible with the right filters.

How do you use tcpdump?

Let’s look at how you use tcpdump. In the examples below, I’m using a Linux VM with one interface, eth0, and the address 10.0.0.8. It has ssh, apache, and postfix services running. Tcpdump requires root access to see the raw packets on the wire, which I will gain with sudo. Be extremely careful who you grant this access to for two reasons. 1) Zombied tcpdump sessions can gobble all the CPU. 2) Since packet contents can be inspected, sensitive information can be seen by anyone with the permission to run tcpdump. This is a security risk, when you must meet PCI-DSS audit requirements. I’ll be using my unprivileged user rnelson0.

Tcpdump by default will try and resolve IP and service names. This can be slow, as it relies on DNS and file lookups, and confusing as most people will search by the IP addresses. We can disable these lookups by adding the n flag to the CLI, adding one instance for IPs and one for services, -nn. We also want to specify the interface, even on a single-NIC node, as it may default to the loopback instead of the ethernet interface, using -i <interface>. This gives us a default argument string of: -nni eth0 or -nni lo, depending on which we are looking for.

Next, we need to generate a filter to look at traffic. The tcpdump man page provides a lengthy list of filter components. One of the most common components is src|dst|host <scope>, which filters for packets from, to, or bi-directionally for the specified IP or network. Others are port <portnumber> and <protocol>, like icmp or gre. We can combine individual components with standard logical operators like and, or, and not: filter for non-ssh traffic to/from 10.0.0.200 with host 10.0.0.0.200 and not port 22.

As a “bonus”, when you run tcpdump with a bad filter, it will exit immediately. It doesn’t offer hints on how to fix the error, but it does let you know right away.

We put this together with the full command tcpdump -nni eth0 host 10.0.0.200 and not port 22. If we ssh to our node and just run this, we won’t see anything happen right away, but we’ll eventually see some ARP packets:

[rnelson0@kickstart ~]$ sudo tcpdump -nni eth0 host 10.0.0.200 and not port 22
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
22:05:50.309737 ARP, Request who-has 10.0.0.1 tell 10.0.0.200, length 46
22:05:50.589052 ARP, Request who-has 10.0.0.253 tell 10.0.0.200, length 46
22:05:59.934464 ARP, Request who-has 10.0.0.200 tell 10.0.0.253, length 46
22:06:51.315637 ARP, Request who-has 10.0.0.1 tell 10.0.0.200, length 46
22:06:51.519754 ARP, Request who-has 10.0.0.8 tell 10.0.0.200, length 46
22:06:51.519807 ARP, Reply 10.0.0.8 is-at 00:50:56:ac:f2:f7, length 28

Now if we view a file on the web server, we’ll see a three way handshake followed by a few PSH packets:

22:17:29.686840 IP 10.0.0.200.59916 > 10.0.0.8.80: Flags [S], seq 1113320281, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
22:17:29.687041 IP 10.0.0.8.80 > 10.0.0.200.59916: Flags [S.], seq 741099373, ack 1113320282, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 5], length 0
22:17:29.690439 IP 10.0.0.200.59916 > 10.0.0.8.80: Flags [.], ack 1, win 256, length 0
22:17:29.690475 IP 10.0.0.200.59916 > 10.0.0.8.80: Flags [P.], seq 1:412, ack 1, win 256, length 411
22:17:29.690540 IP 10.0.0.8.80 > 10.0.0.200.59916: Flags [.], ack 412, win 490, length 0
22:17:29.693772 IP 10.0.0.8.80 > 10.0.0.200.59916: Flags [P.], seq 1:151, ack 412, win 490, length 150
22:17:29.694090 IP 10.0.0.8.80 > 10.0.0.200.59916: Flags [F.], seq 151, ack 412, win 490, length 0
22:17:29.696030 IP 10.0.0.200.59916 > 10.0.0.8.80: Flags [.], ack 152, win 256, length 0
22:17:29.700858 IP 10.0.0.200.59916 > 10.0.0.8.80: Flags [F.], seq 412, ack 152, win 256, length 0
22:17:29.700893 IP 10.0.0.8.80 > 10.0.0.200.59916: Flags [.], ack 413, win 490, length 0

For comparison, here’s what HTTPS looks like when HTTPS is not enabled. You see the SYN packet from the client, and the RST packet comes from the OS since there’s no service listening there:

22:18:50.057972 IP 10.0.0.200.59917 > 10.0.0.8.443: Flags [S], seq 825112119, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
22:18:50.058088 IP 10.0.0.8.443 > 10.0.0.200.59917: Flags [R.], seq 0, ack 825112120, win 0, length 0
22:18:50.558200 IP 10.0.0.200.59917 > 10.0.0.8.443: Flags [S], seq 825112119, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
22:18:50.558264 IP 10.0.0.8.443 > 10.0.0.200.59917: Flags [R.], seq 0, ack 1, win 0, length 0
22:18:51.060995 IP 10.0.0.200.59917 > 10.0.0.8.443: Flags [S], seq 825112119, win 8192, options [mss 1460,nop,nop,sackOK], length 0
22:18:51.061065 IP 10.0.0.8.443 > 10.0.0.200.59917: Flags [R.], seq 0, ack 1, win 0, length 0

Summary

I hope this short tutorial helps you figure out when and how to use tcpdump. If you have specific questions, post them in a comment or ask on twitter and I’ll respond.

Standard

Posted by

Posted on

January 10, 2016

Posted under

Opinion

Comments

1 Comment

A Full Stack What?

You’ve probably heard a lot of talk about the term “Full Stack Engineer”. You may even hear that everyone’s looking for one, so you probably want to be one to help your career. A Full Stack Engineer (hereafter FSE) is someone who doesn’t just know their one area deeply, but knows a bit about the rest of the stack. That depth of knowledge varies from very shallow to deep expertise, with the idea that the FSE knows how the different levels of the stack work together so they can make decisions that benefit the entire stack, rather than a local optimization that may harm the rest of the stack. You don’t want someone making an application decision that blows up the storage stack, or vice versa, so this kind of wide knowledge rather than deep knowledge is definitely helpful.

There’s a huge challenge to becoming an FSE, one of which is the sheer amount of layers in the stack to learn about these days. There’s so much to learn that it’s not actually feasible that any one person can learn all those layers deeply enough to really know the full stack. There’s absolutely nothing wrong with taking the journey toward Full Stack Engineer, but I think there’s another worthwhile goal out there:

A quipped about a “Full Stack Human,” a little bit of a tongue-in-cheek response to the overuse of the FSE term, but there’s some seriousness behind it. What it really means is that you should try and be a well-rounded person. In a sentence: Be more than a job.

A job is (hopefully) only 40 hours out of each 144 hour week and 2000 hours out of 8766 hours a year, less than 25% of what you do in a year. Sleeping should take up about 30% more – and it really should, we have to work very hard to not have a perpetual sleep deficit. Many of us will spend some of that remaining time trying to advance our work and careers, which is perfectly fine. This still leaves a lot of time, time in which we can find some hobbies and activities to enjoy so we’re more than just a working machine.

For exercise, I really like playing flag football. There’s a very diverse assortment of players out there and it’s far more entertaining than a treadmill or machine. When I feel creative, I enjoy woodworking. It requires deliberation, planning, and care in ways that my day job doesn’t – well, since I like having all of my fingers, anyway. I really like my sci-fi and fantasy novels, but I also make sure I fit some classics like War & Peace in between them. My wife and I don’t do anything truly adventurous, but we have been fortunate to visit a number of countries and enjoy their different cultures.

These activities gives you depth and adds dimensions to your character. (I realize I’m starting to sound like your parents did when you were filling out college applications, but bear with me a bit longer!) You meet other people and cultures and gain new viewpoints in which to perceive life. For example, in a decade of flag football, I’ve learned so many different ways to inspire teammates – and which ones don’t work! – and how to calm people down so they don’t lose the game.

I’d never get those experiences just by focusing on working my way up the stack at work, and those experiences help me out just as much at work. We talk a lot about encouraging diversity in tech, and in my opinion, it has to start in your personal life. A well-rounded person, a Full Stack Human, has those diverse experiences and can bring that diversity back into tech.

Your hobbies also give you a healthy escape from work. You aren’t just the project you released last week, and you shouldn’t kill yourself over work (figuratively or literally!). Identification and burnout can be a significant problem for everyone. If you don’t think so, you either aren’t there yet, or you’re there and you don’t know it! When you get too wrapped up in work – the deadlines are pressing down on you, politics got heated, you missed a family event because you were working late and didn’t even realize it – you need a safety valve to relieve that pressure and your personal time should help with that. PSA: If you’re struggling with burnout, please reach out to someone. We’re here to help!

Be a Full Stack Human. I guarantee it will be rewarding on its own, and it’s a huge step up on becoming a Full Stack Engineer!

Standard

Posted by

Posted on

January 6, 2016

Posted under

Opinion

Comments

2 Comments

2015 Recap: How did I do?

Just like I did at this time last year, it’s time to take a look at my goals for the previous year and see how I did.

Learn Ruby

I’d like to think I grok ruby at a more advanced level, now. I’ve written my first gem (and documented the ordeal) and contributed a number of patches to Ruby-ish projects here and there, mostly based around Puppet. I’ve also started writing “throw away” code in Ruby when possible, furthering my transition away from a bash-everywhere mentality. Grade: Pass

Blog more about Security

I started incorporating more security elements into my writing, but I haven’t really done a lot of security-focused writing. I only added one item to the Security category in 2015. I’m sitting on a bunch of drafts about security but am too timid to finish and publish them. Grade: Fail

Home Network

  1. I got my new home network up in running in the late spring, thanks to my partner-in-crime Mike SoRelle. I wrote an article about it as well.
  2. I made some progress here but not in the anticipated direction. I have all of my home network running Linux in Puppet and am working toward the same on the few Windows boxes. There was a lot of turmoil on the VMware side of things (5.5 updates, 6.0, 6.0 updates, changes to VCSA) and it slowed the work there. No IPAM, but I’m not feeling the burn very much because DNS at least is in Puppet.

Grade: Pass. But barely.

Expand PuppetInABox

I’ve learned a lot about software development in the past year. I’ve not only expanded and revamped PuppetInABox (support for Puppet 4 coming soon!), but I’m maintaining a few puppet modules, a puppet-related gem, and am actively participating in VoxPupuli (previously Puppet Community) and contributing features and fixes to Puppet itself. I think I’m making progress here, but still have a ways to go. Grade: Pass.

Propose a PuppetConf Talk

This was originally a goal to propose a VMworld talk, but I changed that as I didn’t have good subject matter for it before the CFP ended. I did submit a CFP to PuppetConf and was accepted! I presented in October and you can catch the video and slides online. I enjoyed the hell out of the conference and I dare say my talk did well, too! Grade: Pass.

VCAP-DCA

I have made zero progress here. It was a busy year! I have until April to get this or renew the VCP and I’m not sure which it will be. Grade: Fail

Read War & Peace

This wasn’t on the list, but it was a personal goal. I’ve read a lot of Barnes and Nobles Classics and I love the Russian literature (Crime and Punishment in particular!), but at ~1100 pages of translated mid-1800 Russian, War & Peace was intimidating. I started this in the latter part of the year and I underestimated the time required to chew on it. I’m around 450 pages in after a few months. It’s been slow but very rewarding. You’ll be happy to know that in 1810, the Russians had meetings about having a meeting. What’s old is new again! Grade: C

I’d say I had a successful year. I didn’t hit all my goals, but like New Year’s Resolutions, I knew some goals would change and others wouldn’t be as important, it was more a guide for the year. I will be posting some new goals for 2016 shortly, though, so I can stay grounded this year as well!

Standard

Posted by

Posted on

December 10, 2015

Posted under

vSphere

Comments

1 Comment

Deploying Windows Images with KMS keys

I’m not all that familiar with Windows licensing models, so I stumbled into a bit of surprise with KMS keys recently. If you are using a central KMS server that you do not maintain, and someone gives you a KMS key, you can ignore it! That’s for the KMS Host, which is where the licensing happens. Your nodes will be KMS Clients and they will use a Generic Volume License Key for activation. The Client communicates with the Host, which tells the client if it is activated and provides all the necessary information for that to happen (I don’t know how the Host does that, that’s the beauty of letting someone else run that service!). In this case, you are often given media to use for the Windows install that includes the GVLK, so you don’t need to do anything but communicate with the KMS Host. It’s a pretty nice setup, all considering.

However, IF you do something silly like put the KMS Host key on your Clients, you won’t get far. The Host key can only be activated 10 times on 6 hosts, so very soon you’ll run into trouble, if not immediately. You have to switch back over to the GVLK and activate using that. Microsoft maintains a list of GVLKs for each edition of Windows. The lookup of the KMS Host is done by DNS, but you can manually configure the KMS Client as well. Once the GVLK is in place, activate the key. Here are the three commands you will need, using Windows 2012R2 Datacenter as the GVLK:

cscript c:\windows\system32\slmgr.vbs /ipk W3GGN-FT8W3-Y4M27-J84CP-Q3VJ9
cscript c:\windows\system32\slmgr.vbs /skms kms.example.com:1688
cscript c:\windows\system32\slmgr.vbs /ato

These commands need to be run from an administrator-privilege command prompt or PowerShell session.

If you are using templates, run the first command on the template. Ensure the deployment process is not adding license information. In vCenter, this means removing all options from the License Information portion of the Customization Specifications in ALL customization specs. Add the /skms and /ato commands to the existing commands in the Run Once section:

KMS Fig 1

KMS Fig 2

When you deploy a VM, it should now automatically activate itself! If you run into issues, ensure that the Client can communicate with the Host and no firewalls are blocking the communication. I’ve found that a global any/<KMS Server>/<kms port> rule in your firewalls is handy to ensure that random networks aren’t blocked from activation.

If you’re interested in learning more about Windows Licensing, Microsoft has a great amount of documentation. I suggest starting with Learn About Product Activation and then moving through the relevant sections.

Standard

Posted by

Posted on

December 4, 2015

Posted under

DevOps, Linux, Puppet

Comments

2 Comments

Kickstart your CentOS Template, EL7 Edition

I wrote an article on kickstarting your CentOS Template in early 2014 that focused on Enterprise Linux 6. Later in the year, RHEL7 was announced and CentOS 7 soon followed. It’s well past time to refresh the kickstart article. To keep this more of a “moving target”, I’ve created a github repo to host the kickstart files at puppetinabox/centos-kickstart, so you can turn there for updates or submit your own PRs. I’m also toying with an existing puppet module danzilio/kickstart that generates kickstart files, and I plan to contribute some PRs to it to manage the kickstart service itself. In the meantime, I’ll show a small profile that will do the same thing, since it’s just apache and a few files.

Kickstart Configuration

The new EL7 file was based off the EL6 version. I simply changed the package list as some were no longer available and the open-vm-tools are now the preferred method of VMware tools management. That section was removed from the bottom. In the additional steps section, I changed the yum repo for puppet from Puppet 3 to Puppet Collections 1 for Puppet 4. I also removed the banner setup, that’s easy enough to add in if you like.

Kickstart Service Management

The kickstart service itself is pretty simple. You can use puppetlabs-apache to install apache and then place your files in it’s default root of /var/www/html. Take the kickstart files and add them to dist/profile/files with any modifications you require. Then create a profile that includes apache plus the kickstart files. That would look something like this:

Continue reading →

Standard

Posted by

Posted on

November 30, 2015

Posted under

vDM30in30

Comments

Leave a comment

vDM30in30 2015 Retrospective

Today ends my vDM30in30 challenge. This makes the 30th post and goes out just a bit before the end of the day on November 30th, 2015. I hit the mark within the timeframe, yay! That’s an improvement over last year’s 25 posts. Writing 30 posts in 30 days was difficult for me, but rewarding. Let’s take a look at why I participated, what I did, and whether it helped me.

I participated in vDM30in30 this year, as in last year, to work on my writing skills. Specifically, I wanted to work on speed. I can write a really long blog post, no problem – some of my Puppet posts were over 5,000 words before I split them up – but it takes me FOREVER! I wanted to work on writing posts of the same length in a shorter duration, but without lowering the quality. This was more than just a requirement to get 30 posts done in 30 days, but something that I think can benefit me elsewhere. Sometimes I spend 10 minutes writing a non-technical email that’s just a single paragraph, and I don’t think that’s really worthy of one sixth of an hour. I was sure I would gain in other ways, but everything was secondary to speed.

Well, not everything. Right before the challenge started, I joined the other participants in trying to encourage others to participate in the challenge. We succeeded, as we had a number of new participants in this year’s challenge! I’ve also spoken to a few people who missed out on the challenge but don’t want to wait until next November to participate, so they may be looking at running the same challenge in January! If anyone else is interested in joining them, let me know in the comments. Thanks to everyone who participated in the challenge, new and existing participants, it was great to see this grow year over year!

Now, back to my challenge efforts. To work on speed, I used a number of tactics:

  • Varied topics. Much of my blog content is what I would consider deep technical content. I wasn’t certain that increasing speed here with the given timeframe was feasible, but I was certain that I could improve speed on this content if I improved speed on other content. I wrote about vSphere, Puppet, Travis CI, and Ruby bundler (all in a not-quite-as-deep manner), and I also branched out into an ode to snow, thoughts on Footloose and 2112, troubleshooting, note taking techniques, our pug Loki, and even got meta about post quality and what to do when the well runs dry.
  • Make November a month of projects. I participated in the challenge while continuing work on other projects (upgrading modules for puppet 4 support, learning Travis CI, Commitmas), making each of these projects fodder for vDM30in30. This ties into the next item, as the project milestone often came under the same time limit.
  • Set a (soft) timer. I often did this by deciding that I had X minutes available, I had a topic I thought could be done in X minutes, and I’d write and post it immediately. I gave myself enough time to do proofreading but I tried to keep to whatever time limit I set. Sometimes I’d have to stop writing because I had to leave the house, and hitting the Post button was difficult but necessary. Of course, I still wanted to keep the quality up so I reserved the right to not hit post or ditch the post entirely. I only made use of this once, and I just needed 5 minutes for proofing when I got back to the computer.
  • Use brainstorming sessions. My normal technique is to think of something I want to write about and then do it. Instead, I would spend 10-30 minutes thinking of what I wanted to write about and making a list in Evernote. By making the list ahead of time, I had a number of solidd ideas to toss around in my head for a few days. When I sat down to write, I often had a rough outline or a list of points to emphasis already. This became especially important at the end of the journey when I started to run out of ideas. If I was going to rack my brain, I wanted to do it for 5 subjects, not just one!
  • Press the post button! Of course, none of the above techniques mattered if I didn’t post the article. I didn’t schedule a single post, every article was made live the moment it was finished. Getting over the fear of hitting post quickly became a secondary goal.

So, did this help me, did I achieve what I set out to? I hit the mark of 30 posts in 30 days and I certainly feel like I improved. I know that I’m proud of myself for following up on my pledge! But did I improve my speed while maintaining or improving the quality of my content? I need to hear from you! I appreciate any and all feedback here in the comments or on twitter. Thank you!

You can see all of the vDM30in30 posts here, including those from 2014.

Standard

Posted by

Posted on

November 30, 2015

Posted under

Puppet, vDM30in30

Comments

6 Comments

Minecraft module for Puppet

At PuppetConf, I had the pleasure of meeting Bren Briggs, who I knew from twitter and IRC, so I was pretty happy when he asked me if I wanted to work on a Minecraft module with him. Of course we’re busy with life and work and the holidays, so we haven’t started yet, but we’re going to try soon!

Bren floated some ideas past me and one of the big questions was, do we want to deploy the Minecraft instance as if we’re on bare metal or via Docker? He started a twitter poll, but those things only last 24 hours and we only received 3 votes. If you were going to use a Minecraft module, would you want one that uses Docker or one that does not?

I’m a little biased here, but I hope people want to see Docker. I’ve not used Docker in anger before, and I need more motivation to update my kickstart setup from EL6 to EL7. But don’t let me sway you, let us know what YOU would like to see. Thanks!

Standard

Posted by

Posted on

November 30, 2015

Posted under

Puppet, vDM30in30

Comments

5 Comments

#Puppetinabox moving to v4 in 2016

I recently used Travis CI to help me get all my puppet modules and my controlrepo ready for Puppet v4. I have one dependent module (ajjahn/dchp) that needs a few polishing touches (issue #7) and then I plan to start moving PuppetInABox to version 4 as well. There are many moving parts but I would like to get this done in the first quarter of 2016.

One thing I will need to do is convert the master’s service from apache/passenger to puppetserver. Unfortunately, stephenrjohnson/puppet does NOT support puppetserver or puppet v4 yet. There are a number of forge modules that provide some level of support for puppetserver and I could use your help in finding the right one. It would seem the clear winner, at least by downloads, is camptocamp/puppetserver. Maybe one of the others is better. Perhaps the stephenrjohnson/puppet module is nearing readiness for version 4.

What are you using, and what tips do you have for someone converting from version 3 to 4? Drop me a line in the comments or reach out on twitter. Thanks!

Standard

Posted by

Posted on

November 29, 2015

Posted under

vDM30in30

Comments

Leave a comment

Success with a first year flag football team

I have been playing flag football for almost 10 years now and I enjoy it immensely. Last year, when we moved to Indianapolis, I was dismayed that I could not find a competitive league to play in. There are a number of weekly pick-up games, though, and through them I learned of a men’s church league and was invited to join a new team there.

New teams can be tricky. A “new” team that’s a split of an existing team with a few new players can often go far, they have a core that’s intact and capable. I ended up on a real new team, where only two members had played together previously and some had never played flag football before. Flag is quite different than regular football. There’s no downfield blocking, you can’t stiff arm, you can’t tackle. All of those lead to penalties or ejections and are often what trip up people new to flag rules, even if they’ve played some HS, College, or even NFL ball. You also lack any connection between players. The quarterback might under- or overestimate a receiver’s speed and end up throwing lots of picks.

For this reason, it’s no surprise that many new teams go 0-X, rarely going 1-X, and almost always losing badly in playoffs if they make it there. In the regular season, we went 1-6. We ended up losing our last regular season game to the #1 team 66-24. Ouch. That put us in last place, meaning we got to play them again the next week in the first round of the playoffs. It would be easy to foresee the loss and throw in the towel right away. I’ve seen lots of teams play a man down due to low attendance or even forfeit that game, knowing it would be a blow-out. Can’t blame anyone, you don’t get paid and some people drive an hour each way plus the game time.

Instead, our team stayed with it. We had more people show up than the previous week. We analyzed what went wrong and came up with a game plan. The opponents used a quick strike offense with timing routes, so we rushed often and played press. By halftime, their QB had thrown 3 picks and we managed to keep it tied. We ended up winning 28-27, scoring the go-ahead points with 12 seconds left. We surprised everyone just by showing up and wowed them by beating the #1 seed.

The next week, we had a matchup against the #2 team. We had the same number of people show up and we again put a game plan together. We had not played the team the week before so our game plan wasn’t quite as good. We went on to lose by one score, 47-42, bringing the team down to the wire.

It was a hell of a season. It was fun to fight adversity and come together as a team to victories, both real and moral. I can’t wait for the spring season to see where we go from here!