At UCSC, NLP MS students design, create, implement, and present NLP Capstone projects to address real-world NLP topics. In a 15-unit course sequence taught over two quarters, students work on teams mentored by industry professionals and UCSC faculty to develop an innovative response to current NLP issues. In many ways, this experience is similar to an internship at a tech company where students demonstrate their technical expertise and leadership skills with mentorship from seasoned industry experts.
In this post, NLP students Sugam Garg and Brian Farrell chronicle their experience in NLP 271A, the first course in our Capstone sequence. Check out what they’re learning about themselves, NLP research, and building NLP solutions as part of their NLP Capstone Experience.
I’m Brian Farrell, and I’m a current Natural Language Processing (NLP) MS student at the University of California, Santa Cruz. As part of my Ambassador role in the NLP Program, I’ve written this post about my experience during my NLP Capstone project. This post also includes insights from one of my Capstone project teammates, Sugam Garg, who also started the NLP Program in Fall 2022 and joined the program with some prior professional experience — you’ll learn a bit more about Sugam’s background in a moment.
From Dublin to Silicon Valley: Brian’s journey to study NLP
To begin, here’s some background about myself. I did my undergraduate studies at Trinity College Dublin and graduated in 2022 with a bachelor’s degree in Computer Science. After graduation, I travelled straight to UC Santa Cruz for my master’s. During my undergrad, I took an introduction to Artificial Intelligence (AI) class and an introduction to Machine Learning (ML) class. Apart from those two classes, I had no other exposure to the world of AI and I had no specific knowledge of NLP. Despite this, I was still interested in learning more about ML, and I decided to apply to UC Santa Cruz to pursue a master’s degree. When I heard about the NLP Program, it immediately caught my attention. And after some research, I felt like it had a lot to offer me – so I applied and was accepted. During the first two quarters of the NLP MS Program, I took a range of classes in ML and NLP. These classes have provided me with a strong foundational understanding of the field, which has prepared me for tackling complex ML tasks and projects.
Accelerating a career in NLP: Sugam’s motivation to join UCSC’s NLP Program
My Capstone project teammate, Sugam, has more than four years of experience as an NLP engineer where he first established his skills in addressing NLP problems including classification and semantic search to production in both cloud and edge environments. Sugam is also proficient in ML and deep learning methods using Python and C++.
So now that you know a bit about Sugam and me, the rest of this post will be structured as a series of interviews recorded at key points during the quarter. In each interview, Sugam and I respond to frequently asked questions about the NLP Capstone Experience to help you get a sense of what it’s like exploring NLP research, picking a project topic, joining a Capstone team, and developing a project proposal.
Sugam, Brian, and Apala meet with NLP Program Director, Professor Ian Lane, to discuss their Capstone project
Week 3 in NLP 271A: Exploration Phase
1. How would you describe your experience so far in NLP 271A?
Brian: We have only just started the first course in the NLP Capstone sequence, NLP 271A, and so far we have been mostly working on paper reviews. We have been divided up into groups and every week each group presents an overview of two papers related to a potential Capstone project topic. We have presented two papers so far and it has provided us with a lot of information about the topic areas of Semantic Role Labeling and Question Answering. We also get to listen to other teams present their papers, which has helped my understanding of NLP research.
Sugam: The discussions during the paper presentations have been very engaging. While each group reviews only two papers, after the discussions the class comes out with a fair idea of the advantages and disadvantages of six papers on one topic. This reduces the individual effort while achieving the same goal collectively.
2. Describe your experience reviewing key literature in the field of NLP. What have you learned so far?
Brian: I have gained an understanding of a range of topics and feel more prepared to pick my Capstone project. Listening to other teams’ presentations has also helped me cement my understanding of each area. The paper that stood out to me the most was titled, “Optimizing Prompts for Text-to-Image Generation.” This paper led to an interesting class discussion about AI-generated artwork and the ethics of training models using the work of artists who do not consent. Personally, I was of the opinion that such models should never use anyone’s work without implicit permission. Reviewing papers as part of a team has been a somewhat challenging but good experience. It can be difficult to organize the team around everyone’s busy schedules, but it is also rewarding to be able to discuss the paper with the group.
Sugam: I have read six papers so far on topics such as Reinforcement Learning for prompt engineering, Semantic Role Labeling, Automatic Speech Recognition, hallucinations in Large Language Models, and Retrieval Augmented Generation. Reviewing these papers is interesting because a lot of these techniques continually reuse foundational concepts and merge them with a novel insight to solve an existing problem. For instance, we read a paper that reused the concepts of simple token classification and mask denoising in a novel way for solving the problem of hallucination.
I am still trying to figure out the best way to work as a team. It has to be a balance between individual contributions and group contributions, and that balance is different for each person. Having four years of industry experience has helped me in trying to take initiative in the group. I organize the discussion around planning and task assignment, and usually all my peers are able to work on items that interest them. This has led to a strong group bonding, something which helps us in supporting each other. It has been a great learning experience to work as a team and collaborate with others who bring a different perspective to each task.
3. Are there some topics in NLP you didn’t know a lot about before this class? What new insights have you gained or would like to explore further?
Brian: A topic we encountered that I was unfamiliar with was Semantic Role Labeling (SRL). One of the first papers we read was on this topic, but it did not introduce or explain the concept so I had to do some research online to understand SRL before I could actually understand the paper. Now that I’ve conducted this background research and reviewed the paper in class, I feel like I have a much better grasp of this topic. I would love to explore this topic further. Even if I don’t end up doing it as part of my Capstone project, I plan to look into it on my own time.
Sugam: Reinforcement Learning (RL) is one topic that I didn’t know much about. It’s an interesting technique that recent Large Language Models have used to enhance their performance. By reading this paper, I found out new methods to combine different reward models for RL. But I still have to study different RL training strategies like actor critic to be able to use it in my projects in the future.
It has been a great learning experience to work as a team and collaborate with others who bring a different perspective to each task.
Sugam Garg
NLP MS Student, 2022-23
4. What’s next in NLP 271A? What are you most looking forward to as part of your NLP Capstone Experience?
Brian: Next we will work on a mock proposal for a Capstone project that will allow us to prepare for our real proposal. The goal of this assignment is to allow us to practice writing proposals and gain some useful feedback. The thing I am looking forward to the most is getting started with the actual Capstone project.
Sugam: I am looking forward to getting my hands dirty with the actual implementation of a Capstone project. The projects seem interesting and I hope to make an impact with my knowledge and efforts!
Week 5: Picking a Project
1. Now that you’re midway through the quarter, you’ve had an opportunity to learn about each of the Capstone projects proposed this year by industry experts and then share your preferences for the projects you want to work on. Share with us what this process of project consideration and selection entailed. What factors were important to you when indicating your project preferences?
Brian: Each week we reviewed papers relevant to two of the proposed Capstone topics and presented our understanding to the class. We did this for three weeks until we had covered all of the topics. This gave me a lot of insight into the topic areas and the current state of the art. Reading these papers helped me gain a better understanding of the proposed projects and what will be expected of me. When selecting a Capstone project, the most important consideration for me was working on something I find interesting as I think it is much more motivating when you are working on a project you actually enjoy!
Sugam: As of today, we have read over ten papers across six different topics. These papers provided a brief insight into these projects and helped me find my interest. Some of these papers also had ideas which we could use interchangeably across different topics. I actually found all the projects interesting, and my preferences were based primarily on a biological motivation of keeping the memory and reasoning part separate. A lot of AI innovations have been loosely motivated from the workings of our brain, and I just found the memory and reasoning segregation pretty interesting!
Sugam, Brian, and Apala joined the 2023 Capstone project team mentored by Dr. Bing Liu, NLP/AI Engineering Manager at Meta
2. Which Capstone team did you both join? What NLP, AI, or ML topics are you most excited to work on as part of your project?
Brian: I am working on the topic “Retrieval-Augmented Generation for Document Question Answering,” which is actually the same project as Sugam and our classmate, Apala Thakur. I am looking forward to the project as I believe there is huge potential in retrieval models and they may be the future of improving Large Language Models. Our mentor also has an interest in creating something product-focused at the end of our project, which I find exciting.
Sugam: I hadn’t worked with Brian in a class project before, so I was very excited to collaborate with him as part of a Capstone team. I always think of AI or software as tools to make tasks easier and scalable. And hence, I brainstormed a couple of ideas to incorporate retrieval models into a product. Overall, I believe we have ambitious goals and it will be exciting to see how much we accomplish!
3. What have you learned so far from your Capstone Mentor and from their experience in the NLP field and in industry?
Brian: Our mentor is a researcher currently working for Meta. So far we have only had two meetings to discuss the direction and background work for the project, and as such we haven’t learned much from him yet. However, I am certain we will learn a lot from Dr. Bing Liu as the project develops. Going forward, we will be meeting with him once a week to discuss the current state of the project and to gain advice on anything we are struggling with.
Sugam: With the ideas we presented to our Capstone Mentor, it was great to see how he breaks the problems down into simpler tasks. He also appreciated some of our ideas and gave us good feedback and it showed his way of thinking. Going forward, I hope to discuss more ideas with him and absorb his process of thinking about projects in general.
4. As part of NLP 271A, you and your team will need to develop and pitch a proposal for your Capstone project. How are you drawing on your prior experience and other NLP classes to inform how you develop a project proposal as part of a team?
Brian: The main resource I plan to use when writing our proposal is a sample project proposal that was provided to us in a previous class last quarter. In NLP 244, Advanced Machine Learning for NLP, we were required to write a proposal for our end-of-quarter project and the professor provided us with a detailed sample for writing good project proposals. This resource will be a great starting point for our Capstone project proposal.
Sugam: NLP 244 really helped in honing my presentation skills. The instructor of the course, Delip Rao, gave us feedback on our presentations and proposals that was extremely thorough and detailed. I am also learning from my participation in the Alexa Social Bot challenge on how to write proposals and elevator pitches. The challenge team is truly amazing and accomplished, and I hope to learn a lot more about project execution from them.
Week 9: Presenting Project Designs at the NLP Capstone Proposal Day
1. Congratulations on completing your first class in the NLP Capstone Experience and presenting your project proposal to NLP Industry Advisors and faculty! What was it like presenting at the NLP Capstone Proposal Day?
Brian: The presentation day went smoothly, and I thought it was a good experience. Throughout the quarter we have done weekly presentations that really prepared us for the proposal. Presenting to the board felt a little intimidating at first, but once we got up and started speaking my nerves went away pretty quickly. The board didn’t have any questions for us so I feel we did a good job presenting!
Sugam: The presentation day was nice – we combined all our efforts from the quarter into one single proposal. The past feedback helped us in creating a thorough and detailed proposal which was appreciated by the industry professionals. We did not get questions, but we did get suggestions from industry experts on things we could consider while working on the project, which was reassuring.
2. How did you apply the skills you’ve gained in the program, your learning from the papers you’ve read, and insights gained in group discussions toward developing your proposal?
Brian: The weekly presentations this quarter definitely helped prepare us for presenting to the board. We have also done several group projects during the program which helped me refine my approach and organizational skills when it comes to working with a team. For writing the proposal, we have done several proposals for previous classes which helped a lot.
Sugam: We used the papers we read to combine ideas in different applications to solve the problems in our project. Exploring a wide variety of techniques was helpful for us in crafting a novel proposal. Our group discussions helped us clarify and refine our proposal so that we have a solid foundation to begin working on.
3. How do you envision you’ll apply the skills and knowledge you’ve gained in NLP 271A in your future career as an NLP professional?
Brian: NLP 271A helped me refine my presentation and public speaking skills which will be useful in future work when I find myself having to present my ideas. I also learned a lot about researching and refining an idea in an iterative fashion, reading papers, and identifying new potential areas and questions.
Sugam: I think the art of crafting a problem statement and a thorough proposal is something that I will carry with me during my professional journey. Our mentor helped us think not in terms of datasets and models but in terms of problems to solve. This development of a thinking process will help us in taking a step back from tools/technologies and allow us to think about actual problems!
Apala, Sugam, and Brian meet at the Silicon Valley Campus to work on the next phase of their NLP Capstone project
The Next Phase: Developing and Implementing an NLP Capstone Project in NLP 271B
What’s next for you and your Capstone team?
Brian: We will take a short break from now until the start of NLP 271B. Once the class starts, my Capstone team and I have a plan and a timeline for implementing our project. We will continue to meet weekly to discuss progress and make any necessary changes.
Sugam: We have set a timeline during our proposal on how we’re going to execute our project. We will read relevant papers during the break and start working on setting up the baseline for our project when the Summer Session starts.
Stay tuned to learn more about Brian and Sugam’s NLP Capstone Experience when they complete NLP 271B in September 2023. Visit our NLP Capstone Projects page for updates from Brian, Sugam, and Apala’s team.