Problem reading rotary encoder with atmega8 - c

Question

I am trying to read a rotary encoder as part of a larger project. I am using a Atmega8L and MPLab X with the XC8 compiler.
The issue I am having, is that the encoder does not work reliable and only seems to be able to count up and not down.
Due to some other aspects of the circuit the encoder pins could not be connected to interrupt-capable pins of the atmega8.
Here is my code, that reads the encoder.
#include <xc.h>
#include "pinmapping.h"
#include "main.h"
static inputEvent lastEvent;
static int aLastState;
static int aState;
//get the last Events data for use elsewhere
inputEvent getLastInputEvent(){
return lastEvent;
}
void initEncoder(){
//get our current button state and set rotation change to 0
if((PINB & ENCODER_SW) == 0){
//also save that to our Event
lastEvent.buttonPressed = 1;
}
else{
lastEvent.buttonPressed = 0;
}
lastEvent.rotationChange = 0;
//also save the current state of the A pin for the first read
aLastState = (PINB & ENCODER_A);
if(aLastState >= 1){
aLastState = 1;
}
}
//this should run regularly eg. on a timer interrupt
void readEncoder(){
//first get the state of both pins
aState = (PINB & ENCODER_A);
int bState = (PINB & ENCODER_B);
if(aState >= 1){
aState = 1;
}
if(bState >= 1){
bState = 1;
}
//check if the button is pressed
if((PINB & ENCODER_SW) == 0){
//save that to our Event
lastEvent.buttonPressed = 1;
}
else{
lastEvent.buttonPressed = 0;
}
//check if the state has changed since last time
if(aState != aLastState){
//check if A and B have the same state -> positive rotation
if(bState != aState){
//save the rotation change to our Event
lastEvent.rotationChange = 1;
}
else{
lastEvent.rotationChange = -1;
}
//save the current state of the A pin for next time
aLastState = aState;
}
else{
//if no rotation change happend, save that to our Event
lastEvent.rotationChange = 0;
}
}
The button press is working properly, but the rotation change is never -1 and always 1 or 0.
The ENCODER_SW, ENCODER_A and ENCODER_B are bitmasks, that are all 0s and a single 1 at the bit corresponding to the correct IO-pin.
In the main, first the initEncoder is called (after IO-setup) and then in the infinite loop a function is called, which in turn calls readEncoder() and then getLastInputEvent() and then does some more stuff before returning to the main.
I have also tried running the readEncoder() function on a timer interrupt roughly every ms. This however results in it not working at all.
What is the problem? Why isn't it working?
Edit: Updated the code after I made some changes.
I am using a ALPS EC12E with pushbutton.
The rotationChange is used to increment/decrement a number that is printed to an LCD.
Edit 2: I now think, it is a timing problem, since I tried running basically the same code, just without anything else on an Arduino and printed the values to the serial console and it worked basically perfectly.
I checked what else is done and realized that in my current code the readEncoder() function only runs about once every 98 or so ms.
When I run the function on a timer interrupt every 1 or 2 ms, it just doesn't work at all. I don't even get a reaction to the button push anymore, which is something I do not understand at all.
In fact the timer does not seem to run at all, or at least the interrupt service routine is never executed. (I tried turning on an LED in the setup and turning it off in the isr, but it stays on indefinitely.)
So I guess theres a new problem now...
Here is the timer initialization:
ASSR = 0x00; //make sure we are not in asynchronous mode
OCR2 = 3; //Output compare register
TCNT2 = 0; //Reset counter to 0
TCCR2 = 0b00001111; //Timer 2 CTC mode, clkio/1024
This is done inside of initDecoder(). The OCIE2 bit is already set earlier in the main setup, enabling the timer 2 interrupt.
The isr than just looks like this:
void __interrupt (TIMER2_COMP_vect_num) timer2_isr(){
//Just read the encoder
readEncoder();
}
The interrupt flag gets cleared by hardware when the isr is executed.
The TIMER2_COMP_vect_num is definitely correct as well, as MPLab recognizes it and I can even look at its definition.